Watch the video at https://youtu.be/zO1p4cPg_yk for client experience and https://www.youtube.com/watch?v=SpN6xKvSgWE for importing experience. Watching these two videos should take 2 minutes total and will greatly increase your comprehension while reading this readme.
MIT for all branches. See the license file in this directory.
Jason Zhang (right): junior at Eastlake High School, developed the cloud-based solution
Google Cloud Functions
Firebase Storage Buckets
Google Cloud Auto-ML
Github Actions for CI/CD
React-Native for UI
Training image classification models is tedious. A user first has to gather images from a phone or camera, figure out how to send those images to a computer, download a labeling app for desktop, export the labeling data into a proper format, and finally begin training.
With our app, all of this is taken care of. A user simply takes picture on their phone, labels them straight on their phone, and sends off the data to be converted into a proper format without leaving the app.
| Overview of our cloud-based solution |
 |
- ✅ A user downloads our app
- ✅ A user takes pictures from their phone
- ✅ A user imports images into our app
- ✅ A user draws bounding boxes through the app
- ✅ A user selects to export their labeled data
- ✅ The user data is send to a firebase cloud function
- ✅ The firebase cloud function forwards image data to firebase storage buckets
- ✅ The firebase cloud function assembles an auto-ML formatted CSV file with references to the bucket and data
- ✅ The firebase cloud function returns the csv file to the user
- ✅ The user can drag and drop the csv file to their autoML console which will begin traning
The data stored in the storage bucket has public read acesss avaliable so the user will be able to import the csv file for as long as the data remains in the cloud bucket.
To simply the making of the app, we employed redux state managment along with react hooks.
- As a user drags his finger to draw a bounding box, the current action data field is changed which is reflected by the changing bounding box on the screen.
- When the user finishes dragging his finger, his changes are pushed to the all_bbox data field
- When a user navigates to a new image, the labeling data is stashed to export_data and a new imaged is poped from avaliable_images.
Employing redux state managments greatly reduces the number of nested components. Using redux hooks, each component can live as a function in its own file and can hook into redux state as its needed. With this model, code reuse and readability is improved.
- We wanted to make the experience fully automatic 🤖 but didn't have time to figure out how to automate the begin-training part with Google AutoML. In the end we decided that this feature isn't actually needed because the app already has pretty low friction and we would't want to pay for someone else's data training.
- For some reason the API crashes 💥when png images are uploaded instead of jpg images. We need to figure out what is up with the base64 to image converter library we are using.
We would like to add an email 📧 API so that a user would recieve an email with their generated CSV file upon posting to the server. We didn't have time to do this. Currently, the user just recieves the CSV file back in plain text that they can copy.Upon choosing to export, the app opens up an email client with the csv link so that the user can easily email the link to themselves. This is no longer a problem.- We had difficulty initially sending a request to our cloud function. It turns out that GET requests cannot carry a body message so we just needed to switch to POST requests.
Sometimes uploading to the firebase storage buckets gets messed up. We are still working on increasing the consistency and speed.this has been fixed with promises but speed can still be increased.The generated csv file contains a blank line at the top which causes the Gcloud autoML parser to crash. Working on a fix.This has been fixed.- The end goal is a service similar to Amazon Mechanical Turks but on mobile making it easy to use for everyone.
| Page on loading of app | Page after selecting an image to label |
 |
 |
| A user importing images | A user labels a picture with oranges |
 |
 |
You can test the API itself yourself by posting data to the cloud function
https://us-central1-bboxlabeler.cloudfunctions.net/api
Here is a data sample that is generated by the app which would then be posted to endpoint:
{
"image1": {
"base64": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD//gAfQ29tcHJlc3NlZCBieSBqcGVnLXJlY29tcHJlc3P/2wCEAAQEBAQEBAQEBAQGBgUGBggHBwcHCAwJCQkJCQwTDA4MDA4MExEUEA8QFBEeFxUVFx4iHRsdIiolJSo0MjRERFwBBAQEBAQEBAQEBAYGBQYGCAcHBwcIDAkJCQkJDBMMDgwMDgwTERQQDxAUER4XFRUXHiIdGx0iKiUlKjQyNEREXP/CABEIABAAEAMBIgACEQEDEQH/xAAWAAEBAQAAAAAAAAAAAAAAAAAEBQj/2gAIAQEAAAAA1bYOz//EABQBAQAAAAAAAAAAAAAAAAAAAAT/2gAIAQIQAAAAV//EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQMQAAAAf//EACUQAAIBBAEBCQAAAAAAAAAAAAIEAwEFERIABhQVITJBQlFSYf/aAAgBAQABPwDtN4ufUl3t67kukJyaUoycA0ADpT2if25B3/ZHUTYZpIqzOC0kZsnOWx+Uh2jDGPX54KTti6ofuBozsqtRyVA1w3KhmQVqJDnwxrx55y6N2WBezOhHG9HNNLNHoICGf3n/xAAZEQADAAMAAAAAAAAAAAAAAAABAhIAIcH/2gAIAQIBAT8AY7UKFmOZ/8QAFxEAAwEAAAAAAAAAAAAAAAAAABNhMf/aAAgBAwEBPwBVun//2Q==",
"data": [
{
"min_x": 0.5,
"max_x": 0.6,
"min_y": 0.7,
"max_y": 0.8,
"label": "dog"
},
{
"min_x": 0.9,
"max_x": 1.0,
"min_y": 0.8,
"max_y": 0.4,
"label": "cat"
}
]
}
}As you can see, the data describes one image with name image1 that contains the two bounding boxes with labels cat and dog.
The API responds to this request by uploading the provided base64 image to a storage bucket and then returning a csv file.
gs://coronatime-7b908.appspot.com/7e283d64-c87b-49dc-9023-9205c6833a32/result.csv
result.csv
UNASSIGNED,gs://coronatime-7b908.appspot.com/7e283d64-c87b-49dc-9023-9205c6833a32/image1.jpg,dog,0.500,0.700,0.600,0.700,0.600,0.800,0.500,0.800
UNASSIGNED,gs://coronatime-7b908.appspot.com/7e283d64-c87b-49dc-9023-9205c6833a32/image1.jpg,cat,0.900,0.800,1.00,0.800,1.00,0.400,0.900,0.400
If this first line is removed, this CSV file is a valid Google AutoML metafile. The user would see a popup in their app with the ability to copy the CSV file for their own safe keeping. In the future, we would probably ask the user to enter an email, so the CSV file could be emailed to them for the sake of convenience.
The google storage bucket that the images were uploaded to is publicly read-only. This means that the user would be able to upload the returned CSV file to either Gcloud AutoML or Firebase MLkit to begin training their model.
Look around the various branches to find all the code that was written to make this user experience possible.
This branch contains all the code used for the react-native labeling app. The following libraries were employed in the making of the app: react-native for UI, redux for state managment, react-native-image-crop-picker for picking images from gallery, and NativeBase so that the buttons do not look hiedeous.
Find the code at: https://github.com/daniel-sudz/UWB-Hackathon/tree/redux-refactor
This is the branch that contains all the code used for the cloud function and firebase storage buckets.
Firebase Functions
Successful Post Request
Cloud Storage Bucket Layout (using uuid to avoid collision)
 |
 |
Find the code at: https://github.com/daniel-sudz/UWB-Hackathon/tree/firebase
This is the branch for our documentation. We hope that you have enjoyed it and that all the images/videos have worked for you.