Practical Development Path Overview
It is better to run and scale a small solid system than to maintain and drag a large sprawling mess.
Development, Scaling, Monitoring, Hardening
The 4 key work categories are application development (getting things working, adding features, improving UI/UX), application scaling (load balanced app servers, DNS round robin, master-slave databases), application monitoring (detailed logging, server performance history), and application hardening (code refactoring, authentication security, database replication/backups, disaster recovery preparation).
You have to be careful with synchronous response dependencies, especially from external web API providers. In case their APIs are down, you want to keep the connection timeout short in order to avoid web server memory overload on your side. In some cases, critical API requests must have a long timeout; in these cases, you want to look at asynchronous solutions such as inserting a task into a message queue (for some worker to perform asynchronously) and just tell the user in the moment that their request has been successfully received for processing.
Tasks are split across front-end (layout, style), back-end (application script, database), and systems administration (server setup, networking).
Generate a SSH access keypair with this script: RSA 4096-bit Keypair Generation | Bash Script
Spin up a server with your public key, noting the server's own public key fingerprint.
Recommended Operating System: Ubuntu or Centos (use the long-term support versions)
Configure and enable the firewall.
Standard Outbound Settings: Allow All
Standard Inbound Settings: Allow Ports 80 (HTTP), 443 (HTTPS), 22 (SSH)
Install the fundamental packages.
Recommended Web Server: Nginx
Recommended Version Control: Git
Recommended Scripting Language: PHP or Python
Use an application framework like Laraval (PHP front/back full framework, good for general web applications) or Flask (Python back-end microservice framework, good for API services or scaling an existing app with separation of responsibilities).
Recommended In-Memory Database: Redis
Recommended Load Balancer: HAProxy or Nginx
Use containers to scale with rapid, uniform server deployment, which is useful for creating auto-scaling clusters and replacement systems. Containers are distinct from normal server deployment scripts in that containers use server images that are an exact copy of an original server state, whereas scripts generally grab new data from external sources and re-run installation processes.
Use central environment servers to set flexible environment variables across multiple server clusters. To do so, application servers are deployed with an environment reload command to grab environment variables from the environment server cluster by calling the environment cluster's load balancer private IP or private domain name. On application server deployment and after every change of an environment variable, make the application cluster supervisor issue an order to every application server to activate their environment reload command.
Clean, auto-scaling systems can be achieved by combining application microservices made of containers that draw from central environment servers, central log servers, central database clusters, auto-updating load balancers, and an organized networking infrastructure. Be warned; this is a late-game design that must be approached at an appropriate pace based on the scaling trajectory of need and cost.
Zero Down-time Changes
Zero down-time application and database changes must be designed and sectioned into distinct, well-ordered, backwards-compatible updates that allow for the gradual re-coloration of a server cluster.
Application Example: If an application update changes the encoding format of a stored phone number, then the database will soon contain phone numbers encoded with both the old and new format. Thus that same update must also change the decoding process to support both the old and new formats. After the first application update, the database must be manually scripted to transform all remaining old-format phone numbers into the new format. Then a second application update may be rolled out to refine the decoding process to only support the new encoding format.
Database Example: Suppose a database table must be replaced by a new table. If a new table is created then populated by data from the old table, then the new table will miss all the new data generated between the time when the old table data is copied and the time when the application is switched to read from the new table. This simple approach might work for small regional applications when done at zero traffic periods, but high-frequency international applications must use a more refined strategy (the correct way). First, create the new table then make an application update that causes new data to be written to both tables (old and new). Second, manually populate the new table with data from the old table. Third, make an application update to read data from the new table and stop writing to the old table. Fourth, remove and archive the old table. This general procedure ensures continuous, error-free operation across every step of the database change.
Network Control - The infrastructure through which users send and receive data packets. Nameservers, regional ISP lines, network firewalls,
Server Control - The hardware on which your direct or virtual server runs. Shell access, server firewall, system users, file and folder permissions.
Application Server Farm/Standbys
Application Load Balancer
Database Load Balancer
System Status Monitor
See examples below:
API Type: SOAP vs JSON