Label and train

1. Label your first images

Make sure to follow the guidelines (right toolbar), which has the most detailed instructions to walk you through creating your first model.

Your labeling tools and how to use them

Select your first class on the left toolbar. Once you click, you will have three different tools you can toggle between by clicking them: AI Detect, AI Select, and Pen. You'll also now be in drawing mode, so you will now need to hold Space to be able to pan across your image.

AI Detect is the fastest, and you should use it unless it isn't working well for you, filling in the gaps with AI Select. The Pen is the slowest, and should be used if AI Select is not working for you.

It is expected and normal to have object cut off by the tiles. When the model is used for analysis, it will automatically stitch across tile boundaries. Do not adjust your tile size to avoid this.

Using AI Detect

AI Detect might not work well for harder images - in these cases, you may need to use AI Select heavily. We'll be improving AI Detect over time.

To use AI Detect on a tile, drag a tight box around one example of your object, using the crosshairs to help you. AI will locate the position of similar-looking objects, and also segment them.

Using the filters

After you use AI detect, you'll need to filter the results, using the sliders. The confidence score slider controls the model score threshold at which objects are filtered out. Drag higher to get fewer labels, and lower to get more labels. You can also filter using color similarity. The size range slider has a minimum and maximum upper bound. You can filter out or include larger or smaller objects using these ranges.

Using AI Select

To use AI select, "prep" a tile by clicking on it, which should be ready in around one second. You will start seeing predictions in real time when hovering.

Labeling objects with AI

1. Label your object by clicking, or drag a box around it for a more precise label on difficult objects.

2. Refine by clicking. Click outside the object to grow your object, and inside to shrink your object.

3. Press A on the keyboard to start a new object.

Using the Pen

Drag and click around your object to get your label.

Select/Delete/Edit

Select

To select one object, click on it on the image, or select it from Labeled objects on the left. To select many, hold Ctrl or Cmd and drag on the image.

Delete

To delete selected objects, you can press Backspace. Press Enter to confirm.

Edit

Make sure that you are out of drawing mode (press ESC). Select an object by clicking it in the viewer or from the Labeled objects section. Then, press the key e or click on the pencil in the Labeled objects section to enter editing mode. From here, you can choose to Add or Subtract from the object. Draw an object that will be added or subtracted from that object. Once your object has been completed, your changes will be automatically saved.

Labeling guidelines

1. Focus on one tile at a time. Label every object in that tile, including objects that are cut off by the edge of the tile. Once every example in that tile has been labeled, click Fully Labeled on top of that tile (in the gray box).

2. Track your progress in the right sidebar tutorial as you mark more tiles as Fully labeled, until you fill up the bar.

3. Make sure you are being careful and consistent with labeling. Don't miss objects, and label the border of objects precisely, moving to the Pen when necessary. It is far better to label precisely than it is to label more data less precisely.

Other settings you might use for labeling

You might need to switch images, add images, configure classes or size groups class, or adjust your brightness/contrast settings. Find the instructions for those below.

Classes

Changing class color

You can also choose from different colors to represent the class. Simply click the colored circle to open up a color picker.

Changing the name or deleting a class

Hover over the class to reveal the 3 dots button, which you can click. From there, use the dropdown to rename or delete a class. If you delete a class, every object created for that class will be deleted as well.

Add a new class

Click the + button next to your size group on the left sidebar to add a new class. You might need to add a new size group if your new class represents an object that is of significantly different size or a different segmentation type.

Size groups

Adding a new size group

Click Add size group from the left bar, and follow the setup flow to select your segmentation type and tile size.

What is a size group and why should I make one?

A size group is a group of classes that use the same tile size and segmentation type. They allow your AI model to see your objects at different scales in the same image, to optimize your accuracy. This also allows you to infer interesting parent-child hierarchy information from very different sized objects.

Create a new size group if you want to:

• Blending region segmentations (confluence, inflammation, tissue regions) with object (cell, cyst, organoid) in one model.

• Blend two very different sized objects in one model (organoid and cells, tumor and immune cell, crypts and goblet cells)

• Separate labeling for sparse vs. dense classes (rare cell type vs. very common cell types)

Keep in mind that adding a new size group increases training and analysis credit usage.

Switch or add images

Switching between images

Click on the dropdown under Choose image on the left sidebar, and select a new image to change to it. You can scroll, filter, and search to find your image. Alternatively, press Next or Prev to move forward or backwards.

Adding images

If you need to upload more images, you'll need to exit the dashboard entirely and add more images. Click the X in the top left to exit, and look for the Add or remove data tab.

Brightness/Contrast

Adjusting brightness/contrast

Note: your model will not see these adjustments! If you need to adjust a channel to be able to see an object, you should make those changes in the project channel settings, as detailed in Project settings

From the top bar, click Settings -> Image settings Here, you can adjust brightness and contrast on a per-channel basis, or hide/show a specific channel for an image.

2. Train your first AI model

Once you've filled up the Tutorial progress bar, you're ready to train your first AI model! Click on the Train tab, and click Train a new model. You'll see the following screen. You can add an optional description and customize transforms if needed. Then, click Start training.

Now, you can monitor your training progress. It will have a progress bar and an approximate ETA.

3. Test and improve your model

Once fully trained, we'll see how well your first model does on tiles it hasn't seen.

Navigate to Project and model details, and click on the AI model version, and click on the Statistics tab. You'll see "Quantitative metrics," These metrics indicate how this AI model performed on the images selected for training and test. If there is a particular group of data your analysis didn't do well on, it might mean you need more data in that area.

"Training Set" refers to tiles that you labeled that were used to train the AI model. "Test Set" refers to tiles that were not included in training the AI model. The statistics given are a measure of how close the AI model was to replicating your labels, that is how accurate it is.

Intersection-over-union (IoU) indicates how closely the AI's predictions are with your training labels you put in. It's calculated by dividing the area of overlap between the predictions and your training labels by the union, that is the area covered by at least one prediction. That gives a score between 0 and 100, where higher scores indicate better accuracy.

The bbox column under statistics means a bounding box around the predicted and labeled objects while the mask means the outline of the detected object itself.

If the score is 100, that indicates the predicted labels were exactly overlapping with the labels you put in, whereas a score of zero means no predicted labels overlapped with your training labels.

This can be helpful in quantifying how accurate a model is, but qualitatively determining if a models is making the right calls or not is often better.

Navigate to Label on the top bar, or click Evaluate on your trained pipeline to be taken there.

Test using AI Prelabel

As prompted by the right toolbar, use the new AI Prelabel button next to your size group to test your new model out. Click the tool, and then click unlabeled tiles on your images to prelabel them - they should be labeled in less than 10 seconds. If you have multiple size groups, you'll need to do this for each size group.

Your first model will almost always have mistakes after your first training run, due to the very small number of objects it was trained on. Choose Improve model.

Improving your model

Now that you've trained a first model, you should probably stop using AI Detect. Using AI Prelabel, followed by AI Select for corrections is more efficient after training your first model.

Once you've trained and evaluated your first AI model, you will now be able to label and improve assisted by AI. Your workflow is now:

1. Prelabel tiles with AI - Use the new AI prelabel tool on the left toolbar to test your model, and generate labels with your last model version.

2. Correct prelabeled tiles - Prelabeled tiles are shown in purple. Correct these by adding, removing, and editing objects, and mark them as corrected. Make sure to focus on one tile at a time, only clicking Corrected once that tile is completely correct. Verify that every object has been labeled, even ones on the edge, and incorrect labels have been deleted.

3. Train and test - Once you have a good amount of newly fully labeled and fully corrected tiles, train a new version like before to get a better model. The tutorial will suggest how many tiles you should correct to get a big boost in performance, but you can do this at any time.

You'll want to repeat these 3 steps until you have perfect or close to perfect performance! In most cases, it's best to improve your model at least two times before moving on, but you can always come back to improve it more.