In this article we’re going to build three projects utilizing AI functions of Codecraft, a graphical programming environment based on Scratch 3.0. Codecraft is developed and maintained by TinkerGen education and is free to use.
AI and it’s subset, machine learning, has become an integral part of our lives in the last 10 years. School curriculums in many countries include AI and machine learning knowledge, but it can be difficult for an average teacher to create lesson materials for teaching this topic without right tools. Using machine learning libraries, even the most user-friendly ones, such as Keras, to build an application the students can interact with, can be a time-consuming and daunting task. This is where Codecraft might help – it has a rich set of machine learning powered applications the teacher can use to explain and demonstrate concepts related to artificial intelligence. Let’s have a look at them!
Step 1: AI Extensions Overview
Open Codecraft online editor at https://ide.tinkergen.com/ .
Click on Stage mode and then to Add Extension Button at the bottom of blocks navigation tab.
You’ll see there are three AI Extensions available:
– Cognitive Services — machine learning models for processing live video feed and sound
– Teachable Machine — can be used for training a classification model with pictures taken with your camera
– Translate — utilizes machine learning for translation
In this article we will focus on Cognitive Services and using these functions to control Grove Zero hardware. Wireless communication between computer and Grove Zero is still work in progress and I will update the article as soon as it becomes available. For now we will use wired connection to pass the messages between Grove Zero module and computer.
Step 2: Play Space Invaders With Your Nose
For the first task we will use only stage mode without any hardware. Codecraft has wealth of examples to get started with programming – one of these examples is programming classic game of Space Invaders in Codecraft Stage mode. We’re going to expand on this example and add the ability to control the in-game fighter with your nose using Cognitive services.
Click on Examples button. Open SpaceInvaders example. We only need to change the part relevant to movement. For that we will get x-position of the nose on the face detected in live video feed, save it in variable pos. Then if value of pos is larger than 50, we move our space fighter to the right, if it is less than -50, then we move fighter to the left. This is it. Try it out! If you feel the fighter movement speed is too fast, you can change change x by … block to a smaller number.
Next we will expand on this functionality to control Grove Zero car and make it move forward/backward/left/right.
Step 3: Control Grove Zero Car With Codecraft AI
Let’s start by writing code in stage mode – this is the part, that is responsible for analyzing the video feed and outputting commands to Grove Zero.
1)We execute the code in the first screenshot until space button is pressed. After that we send stop command.
2) Inside repeat until block we check the x-pos and y-pos of the nose of a detected face and save them to corresponding variables.
3)If absolute value of x-pos is less than 21 (meaning it is between -20 and 20, so near the center of the image), then we go on to check y-pos to determine if car should go forward or back
4) Otherwise we check if car should go left or right
Now let’s move to Device tab and write short code for Grove Zero car. We will use When I receive block, which is responsible for receiving internal broadcast commands. Not to be confused with When radio receives, which is used for Bluetooth communication. Consult the screenshot for details, the code is fairly simple.
Connect your assembled Grove Zero car to computer and press on Connect button in Codecraft. Switch to online debugging mode – that will allow Grove Mainboard to receive broadcast commands over the wire. Then go to stage mode and press on the flag. The car should be moving now, try controlling it with your nose 🙂
Step 4: Line Following With Voice Control
There’s a map in Grove Zero car kit, which is originally for playing Collect the Coins mini-game. We can repurpose it for using voice recognition in Codecraft to give commands to the car on which way to turn on the crossroads.
The stage mode part here is less complicated than in previous example. What we do is:
1) Upon receiving broadcast message choose_direction, recognize speech fragment and save it in result variable.
2) Keep recognizing until result is NOT unknown.
3) Compare result with two strings – “left” and “right”. Broadcast corresponding message to Grove Zero car.
The code running in Grove Zero car is a bit more complicated this time, but essentially it follows this logic:
1) When start, broadcast forward message
2) Upon receiving forward message, start following the line. If line is lost, stop the motors and broadcast
choose_direction. Here the code in Stage mode starts gets executed.
3) If message left received start turning left and then switch back to line following mode.
If message right received start turning right and then switch back to line following mode.
Step 5: Behind the Scenes
The neural network models used in the above application are all run locally in your browser, which has a few distinct advantages as compared to sending the data to the cloud for processing: smaller latency and better privacy. A number of neural networks are used in Cognitive services – Sound Classification for speech commands(, Face Landmark Detection, Face Expression Recognition and Age estimation.
In this article we explored the basic functionality of one of Codecraft AI extensions – Cognitive services. There are multiple ways you can build on these examples to make even more fun and exciting applications!
If you decide to give it a try,be it with Grove Zero or just using Stage mode, do share in the comments below. For more information on Grove Zero series, Codecraft and other hardware for makers and STEM educators, visit our website, https://tinkergen.com/.
TinkerGen has created a Kickstarter campaign for MARK(Make A Robot Kit), a robot kit for teaching coding, robotics, AI!
Source: AI With Grove Zero and Codecraft (Scratch 3.0)
