2021-02-25

Github tag 0.4.0.0

Just getting back into this project after quite a long gap (i.e. more than a year!). In the time between I've got my TypeScriptUtilities project to a reasonable state of completeness... it's now got the basic components I'll need to support client to server error logging and event sourcing in TryML. As such I'm switching the focus of this project from the front-end and UI to the server-side.

Since my last post I've added a new project called 'MnistImageStore', which is a small microservice whose function is to serve up images from the MNIST database... it just exposes a couple of endpoints to allow retrieving MNIST images either by id or by label. The reason for creating this project and the general shift in focus to the server side is that I want to get a basic microservice infrastructure setup, and just establish some patterns which I can then follow as I add more microservices (e.g. for neural network training, weight caching, etc...). Additionally I want to get the CI/CD / DevOps / Deployment (whatever you like to call it) setup while the application is still at a very basic stage... ideally in as much of a platform agnostic way as I can... i.e. ideally I want to have the ability to deploy to multiple cloud services (likely start with AWS and Azure), Docker/Kubernetes, and locally for testing, without changing the core code base.

I've been thinking about how to structure the Visual Studio project to best suit an ASP.NET Core microservice solution... mainly whether each microservice should be a separate project within a single solution, or each should be in its own solution (considering things like referencing shared libraries, easy of debug and test, deployment flexibility etc...). As it stands 'MnistImageStore' is a project within the 'TryML' solution, and I'll add other microservice projects into this solution. Since Visual Studio allows nominating multiple startup projects (in the solution properties), and then multiple projects can launch under IIS, this still allows easy local debugging. For now I'll run with this approach, but reconsider it if any major issues or limitations come up.

A few other things that I've either done, or are on the horizon..

Upgraded to .NET 5
Reviewed Microsoft's REST API Guidelines. These are really good... even if you don't agree with the specifics of everything contained, they're really comprehensive, and cover all the key points and features you need to think about when building a REST API not only as a backend for GUI applications, but APIs which are used programmatically by other APIs and systems.
Looked at error handling... especially for 500 errors... trying to find a nice and consistent way to adequately return all the data contained in internal exceptions that cause the 500s to occur. Microsoft have some recommendations in the aforementioned API guidelines, but there is also RFC 7807... need to decide whether I want to go with one of these, or create my own.
Thought about filtering and searching data returned by API endpoints. The Microsoft guideline recommendations look very similar to OData which I like, but it's also not trivial to implement. I'll potentially start by just custom implementing a small subset of OData (e.g. 'eq', 'gt', 'lt', and basic logical operators)... for example to be able to search and filter the MNIST images returned by the 'MnistImageStore' microservice.
I need to decide on a way to do endpoint versioning.
Need to build a more generic implementation of MnistImageStore.Persistence.IMnistImageReader which can read from a URI, so that the MNIST files could stored in AWS S3 or Azure Blob Storage.
I want to integrate metrics capture into the server-side code while the whole project is still small. I may look at using my own ApplicationMetrics project for that although it may need some changes/tweaks to bring from .NET Framework into Standard.

Anyway, it's nice to be able to focus some time again on a more practical/real-world project (after spending quite a lot of time on class libraries), so am looking forward to the next steps.

2020-02-11

It's been ages since the last time I looked at this project, mostly because I've been busy with work on other things. Since last post I have updated Angular and Material Design to the latest versions (8.2.4 and 8.2.3 respectively), and managed to get Material Design components displaying properly in the UI. This was a bit of a pain, encountering lot of npm UNMET PEER DEPENDENCY errors, and having to try to manually upgrade many dependent npm packages. In the end, since the project is still small, it was easier to just copy the Angular components I'd already created to another folder, delete the whole project UI folder, and install Angular CLI and Material Design again from scratch.

In any case the main purpose of this post is just to get a coherent set of next steps that I should work on. Looking back at last post's TODO list, some of those things have been done... specifically...

I've built the beginnings of a TypeScript error handling library in TypeScriptUtilities, and need to create an implementation of IHttpClient for the Angular HTTP client as a next step.
I've started work on a generic TypeScript logging library (https://github.com/alastairwyse/TypeScriptUtilities/tree/master/src/utilities/logging)
I've got TypeScript unit tests running in TypeScriptUtilities via jest and ts-jest

So the next steps are...

Error Handling

Implement IHttpClient for the Angular HTTP client

Logging

Finish this off in TypeScriptUtilities

Visual Design

Get the basic UI for retrieving and displaying MNIST images looking nicer and using Material Design Theming properly

Deployment

I'll start with Jenkins since I've already used this on previous projects. Initially need to get it to...

Run jest unit tests
Transpile the UI code into a set of static files which can be served up from the web server instance
Run .NET unit tests
Build the .NET project
Create a web server instance (will probably use AWS Elastic Beanstalk in the first instance since I already got this working on a previous project) and deploy / run the transpiled / compiled source

The next, next steps will be...

Service Discovery

I'm thinking about how I can make the deployment of the solution ultra flexible... i.e. have the following 3 (or more) deployment types be able to be pushed out simply via config changes...

Proper 'production' setup running on cloud-based web server
Running on the local machine with different backend services running as separate processes (and potentially interacting through something like named pipes to remove dependency on the TCP stack)
Running on the local machine within a single process... i.e. the different services existing as separate class instances and interacting within the .NET code of that process.

Likely I will want to have a 'service interface' that the front end code interacts with (which will need to serve up REST endpoints for the front end to consume), but behind this some kind of service discovery process should retrieve abstracted implementations of each of the required backend services. E.g. the current backend (providing MNIST image data) could be defined as interface IDigitImageService, which provided methods like GetDigitImage(int imageId). The service interface layer could be responsible for retrieving and storing an implementation of this interface which could...

exist either remotely or on the local machine (and then within the same or a different process)
be implemented using different technologies... e.g. a simple in-memory Dictionary, a local database, a remote database etc...

The service interface and front-end code wouldn't care how or where the IDigitImageService instance was implemented... as long as it provides the data specified in the IDigitImageService interface.

Event Sourcing

Early on (like while there's only a single screen), I want to implement event sourcing so that all user interactions can be captured, and potentially replayed for regression testing purposes.