A lot happened Since I wrote the recent post. The most important aspect is the general availability of an AWS DeepRacer Console. This has very significant influence on my plans.
AWS DeepRacer Console is an Amazon Web Services service consolidating other services. First of all it uses SageMaker RL which is an AWS reinforcement learning solution. Then it uses AWS RoboMaker which allows simulating a physical environment for developing physical machines. Those two services (and a couple more) are integrated together into a single page where user can provide a reward function, select a track to train against, has a chance to update a couple parameters in the reinforcement learning algorithm, pushes a button and can watch streamed visualization of the learning attempt. I don't think it can get any more hands-on than that.
If you managed to plow through to the end of my post, you knew that I planned to learn how to set up the training without the Console. I regretted deeply no access to it but then expected it to become generally available after the summits since it wasn't out there before. Amazon did provide a set of instructions to set up the training of the model but I initially failed to do that one way, the other one seemed rather slow to learn and I did not understand what was happening. I wanted to do it right, first learn and then understand.
Now things have changed.
With seven days left to the summit I will focus my efforts on learning to train and training a model.
I already completed watching the free AWS DeepRacer training which I really recommend as it explains everything I wrote about last time in a much better way and even more. The pace is not to fast, the way lessons are being presented in a non-overwhelming way. You need an Amazon store account to access the training.
To do the exercises you will need an AWS account which you can get here. The Free Tier account provides enough resources to learn many things that AWS offers. Just remember that some of them cost money and it's better to clean up after work to not get into some big bills. Within the first month of a Free Tier account there is roughly 10 hours of free SageMaker training, later it costs money. One thing I do value about Amazon is that they are quite open about what costs you are being exposed to, for instance: DeepRacer Pricing, that said it's good to monitor Billing so that the bill doesn't surprise you. And clean up after yourself.
The whole thing sounds scarier than it really is when you actually try, so I really recommend checking it out.
If you would like to learn more about AWS and their services, there's a number of courses available. I did the CloudGuru AWS Certified Developer - Associate and it was OK. Just enough for me to not fear AWS before the conference and hands-on workshops Again, it costs money, But every now and then a large discount of about 90% is offered. Just to be clear: this is not the only course and there might be better ones out there. I paid for this one and found it useful
DeepRacer Console
Once you visit the AWS DeepRacer Console, you are presented with a short introduction, invitation to get started, to participate in the DeepRacer League.
At first you are presented with three sample models. While they are fun to evaluate, I don't recommend doing anything else with them. The key reason is action space - a set of behaviours that your model is allowed to present. The sample models have their behaviours rather slow therefore I recommend starting from scratch. You can have a look at their reward functions and reuse them.
My first model was built based on a combination of actions from basic and advanced reward function samples. After only two hours of training I got these results:
One thing I've learned from the course is that I should not expect them to transfer directly into the outcomes on the track with a real device, but it IS fast to me.
There are six tracks to choose from when training a model:
I'm expecting more to come as new races are announced.
DeepRacer League - virtual races
I've talked enough about the races during AWS Summits around the world. The winners of those races will be invited to AWS re:Invent 2019 to compete for the DeepRacer Cup. But that's not all: additional virtual races have been started winners of which will also be invited to the finals.
Once you have trained your model, you can submit it to the virtual race track. The trick is that the training and evaluation tracks differ slightly. For the re:Invent 2018 track your training and evaluation happen on the same track. This allows you to introduce optimizations that can overtrain a model to be able to nail this track only but not other. As a result you can complete that track in 8 seconds (click to see a video on Twitter).
When you train your model to have a perfect score on a straight line track, don't expect it to do well on a track with turns. That's why having a model that is trained to behave well in various circumstances can guarantee you a better result.
I have started the training process which will cost quite a bit, but I am hopeful to get some fun out of that. The next step I plan to take is to complete the workshop on GitHub. I want to be prepared to take the most out of the Summit in terms of DeepRacer experience.
I have checked what the parameters provided in a reward function look like:
{
'all_wheels_on_track': True,
'x': 4,
'y': 5,
'distance_from_center': 0.3,
'heading': 359.9,
'progress': 50,
'steps': 100,
'speed': 1.0,
'steering_angle': 6,
'track_width': 0.2,
'waypoints': [[ 2.5, 0.75], [3.33, 0.75], [4.17, 0.75], [5.0, 0.75], [5.83, 0.75], [6.67, 0.75], [7.5, 0.75], [8.33, 0.75], [9.17, 0.75], [9.75, 0.94], [10.0, 1.5], [10.0, 1.875], [9.92, 2.125], [9.58, 2.375], [9.17, 2.75], [8.33, 2.5], [7.5, 2.5], [7.08, 2.56], [6.67, 2.625], [5.83, 3.44], [5.0, 4.375], [4.67, 4.69], [4.33, 4.875], [4.0, 5.0], [3.33, 5.0], [2.5, 4.95], [2.08, 4.94], [1.67, 4.875], [1.33, 4.69], [0.92, 4.06], [1.17, 3.185], [1.5, 1.94], [1.6, 1.5], [1.83, 1.125], [2.17, 0.885 ]],
'closest_waypoints': [3, 4],
'is_left_of_center': True,
'is_reversed': True
}
There is some wiggle room in terms of what we could optimize with such data, I'll be thinking about it this week.
Summary
The two goals I'm hope to fulfill at the AWS Summit in London is meeting wonderful people and having fun. I'm pretty confident about the first one taken into account the folks I've met at the re:Invent.
So head to my previous post on https://codelikeamother.uk or Steemit and watch the videos and follow the links. If you'd like to get in touch, leave a comment below.