A Midway Report on my Computer Vision Project

python
fastai
useful-tools
redactionmodel
A report midway through my computer vision project to detect the presence of redactions on government documents.
Author

Alex Strick van Linschoten

Published

January 16, 2022

Part 6 of 17 Redaction Model
All parts
  1. Training a classifier to detect redacted documents with fastai Sep 2021
  2. How to annotate image data for object detection with Prodigy Nov 2021
  3. What is VFNet? Nov 2021
  4. 73% accuracy for redaction object detection Dec 2021
  5. A Taxonomy of Redaction Dec 2021
  6. A Midway Report on my Computer Vision Project Jan 2022
  7. It's raining bboxes: how I wrote a Python script to create 2097 synthetic images to help improve my machine learning model Feb 2022
  8. Incremental Improvements to my Redaction Detection Model Mar 2022
  9. Figuring out why my object detection model is underperforming with FiftyOne, a great tool you probably haven't heard of Mar 2022
  10. 'I guess this is what data-centric AI is!': Performance boosts after training with synthetic data Apr 2022
  11. How to trust the data you feed your model: data validation with Great Expectations in a computer vision context (part 1) Apr 2022
  12. How to trust the data you feed your model: data validation with Great Expectations in a computer vision context (part 2) Apr 2022
  13. How to trust the data you feed your model: alternative data validation solutions in a computer vision context (part 3) Apr 2022
  14. A painless way to create an MVP demo using computer vision models May 2022
  15. More Data, More Problems: Using DVC to handle data versioning for a computer vision problem May 2022
  16. It takes a tribe: how I'm thinking about putting my object detection model into production May 2022
  17. Storing Bytes: what data serialisation is and why you need it for machine learning Sep 2022

(This post is adapted from a twitter thread, so is a bit more terse than usual.)

I recently switched what I spend the majority of my professional life doing (history -> software engineering). I’m currently working as an ML Engineer at ZenML and really enjoying this new world of MLOps, filled as it is with challenges and opportunities.

I wanted to get some context for the wider work of a data scientist to help me appreciate the problem we are trying to address at ZenML, so looked around for a juicy machine learning problem to work on as a longer project.

I was also encouraged by Jeremy Howard’s advice to “build one project and make it great”. This approach seems like it has really paid off for those who’ve studied the fastai course and I wanted to really go deep on something myself.

Following some previous success working with other mentors from SharpestMinds on a previous project, I settled on Computer Vision and was lucky to find Farid AKA @ai_fast_track to mentor me through the work.

In the last 6 weeks, I’ve made what feels like good progress on the problem. This image offers an overview of the pieces I’ve been working on, to the point where the ‘solution’ to my original problem feels on the verge of being practically within reach.

After just a few lessons of the FastAI course, I trained a classification model to ~95% accuracy to help me sort redacted images from unredacted images.

I used Explosion’s Prodigy to annotate an initial round of data to pass into the next step, enjoying how the labelling process brought me into greater contact with the dataset along the way.

I switched to using IceVision to help me with the more complicated object detection problem, using MMDetection and VFNet to get pretty good results early on.

I’m currently in the process of creating my own synthetic images to boost the annotations I’ve manually made. (I’ll be writing about this process soon as well, as I’m learning a lot about why this is so important for these kinds of computer vision problems.)

I’ve also been amazed at the effectiveness of self-training (i.e. using my initial model in my annotation loop to generate an initial set of annotations which I can easily amend as appropriate, then feeding those annotations in to create a better model and so on). More to follow on that step, too.

I started using Evidently to do some drift detection, inspired by some work I was doing for ZenML on adding Evidently as an integration to our own tool. This helped me think about how new data was affecting the model and the training cycle. I feel like there’s a lot of depth here to understand, and am looking forward to diving in.

I made a tiny little demo on HuggingFace Spaces to show off the current inference capabilities and to see the model in a setting that feels close to reality. This is a simple little Gradio app but I liked how easy this was to put together (a couple of hours, mainly involving some build issues and a dodgy requirements.txt file)

Along the way, I found it sometimes quite painful or fiddly to handle the PDF files that are the main data source for the project, so I built my own Python package to handle the hard work. I used fastai’s nbdev to very quickly get the starters of what I’m hoping might be a useful tool for others using PDF data for ML projects.

Throughout all this, Farid has been patiently helping guide me forward. He saved me from going down some dark rabbit holes, from spending too long studying skills and parts of the problem that needed relatively little mastery in order to get to where I am.

Farid has been a consistently enthusiastic and kind advocate for my work, moreover, and this has really helped me stay the course for this project that takes a decent chunk of my time (especially seeing as I do it completely aside / separately from my day job).

I feel like I’m consistently making progress and learning the skills of a data scientist working in computer vision, even though I have so much left to learn! My project still has a ways to go before it’s ‘done’, but I’m confident that I’ll get there with Farid’s support. (Thank you!)