73% accuracy for redaction object detection

redactionmodel

computervision

progressreport

I made some progress on my redaction model.

Author

Alex Strick van Linschoten

Published

December 11, 2021

Part 4 of 17 Redaction Model

All parts

Training a classifier to detect redacted documents with fastai Sep 2021
How to annotate image data for object detection with Prodigy Nov 2021
What is VFNet? Nov 2021
73% accuracy for redaction object detection Dec 2021
A Taxonomy of Redaction Dec 2021
A Midway Report on my Computer Vision Project Jan 2022
It's raining bboxes: how I wrote a Python script to create 2097 synthetic images to help improve my machine learning model Feb 2022
Incremental Improvements to my Redaction Detection Model Mar 2022
Figuring out why my object detection model is underperforming with FiftyOne, a great tool you probably haven't heard of Mar 2022
'I guess this is what data-centric AI is!': Performance boosts after training with synthetic data Apr 2022
How to trust the data you feed your model: data validation with Great Expectations in a computer vision context (part 1) Apr 2022
How to trust the data you feed your model: data validation with Great Expectations in a computer vision context (part 2) Apr 2022
How to trust the data you feed your model: alternative data validation solutions in a computer vision context (part 3) Apr 2022
A painless way to create an MVP demo using computer vision models May 2022
More Data, More Problems: Using DVC to handle data versioning for a computer vision problem May 2022
It takes a tribe: how I'm thinking about putting my object detection model into production May 2022
Storing Bytes: what data serialisation is and why you need it for machine learning Sep 2022

Last time I wrote about my redaction model training project, I explained how I used Prodigy to annotate and label a bunch of images. I subsequently spent a long evening going through the process, getting to know my data. I managed to make 663 annotations, though quite a few of those were negative annotations: I was stating that a certain document contained no redactions at all.

Once I had my redactions, I needed to convert the files from a Prodigy format into a .coco annotation format. I am using IceVision, a really useful computer vision library, for which it is easier if I pass in the annotations in the .coco format.

From that point, it was fairly easy to follow the steps of the object detection tutorial outlined in the IceVision documentation. I ran into some problems with Paperspace Gradient not easily installing and importing IceVision. For some reason files don’t get unzipped on Paperspace, but it’s possible to just do this manually:

Do the basic install, including the import of icevision.all. Wait for the error to get raised, then open up a terminal and enter:

cd /root/.icevision/mmdetection_configs/
rm v2.16.0.zip
wget https://github.com/airctic/mmdetection_configs/archive/refs/tags/v2.16.0.zip
unzip v2.16.0.zip

Then run it again as normal. Later on, another error will get raised. Fix it with this (again in the terminal):

jupyter nbextension enable --py widgetsnbextension

This enables ipywidgets in the notebook, I think.

Once through all of that, I was able to fine-tune a model based on the annotations which I currently have. I selected VFNet as the model I wanted to use as the pertained model. After training for 40 epochs, I reached an accuracy of 73%:

If we look at some of the results (using model_type.show_results()) we can get a sense of the parts it found easy and the parts which it found hard. (All the boxes below are what it as predicted, not the ground truth annotations.) Some identification of boxes went as you might expect:

I was surprised that something like this worked as well as it did:

It wasn’t perfect, but I don’t remember having annotated too many of this specific redaction type, so I’m fairly happy with how it worked out. You can see it still makes a number of mistakes and isn’t always precise about where the boxes should go. I hope that’ll improve as I add more examples of this type of redaction.

My next steps for this project include the following:

create synthetic data. The redactions are probably easy enough to mimic where we’ll get a lot of value from the use of synthetic data (fake redactions on not-real document backgrounds). It’ll be an easy way to boost my training data set by a good amount, hopefully leading to big improvements in my model accuracy.
potentially add in either active learning (to help speed up my annotation process) or self-training (using the model to make annotation suggestions on unlabelled data and using only the suggestions with really high confidence estimates).
think through the augmentations that I use as part of my workflow. I basically want augmentations that are similar to however the production use case will be: i.e. the kinds of redacted images that it might see when being given real-world data at inference time post-training.
add in experiment tracking. I’ve never used something like Weights & Biases, so I’m excited to try that out and have a real process for tracking my progress throughout this project.
cleaning up and refactoring (a bit) my repository where the code lives for processing the input data. It’s starting to get a bit unwieldy and I’m worried I’ll start to forget the order things were done and some of those small details.

Get new posts by email