Software 101

Summary of Internal Tools

Agenda

Please fill out this attendance form. Completion of this form is to be used to display the demographics of the club as well as record attendance.

This is the agenda for the Software 101 workshop. The workshop date is October 9, 2018. Meeting lasts from 6:00 PM - 8:30 PM. Please note that the minutes are approximations, so expect each concept to have an additional ±10 minutes.

  • Software Setup (20 mins)
  • Bash Scripting (10 mins)
  • Python Scripting (10 mins)
  • Docker (15 mins)
  • Jenkins (15 mins)
  • Feedback Form (5 mins)
  • Reminders
  • Free Time (75 mins)

Description

The Unmanned Aerial Systems club is unique on campus in that we develop all of our internal tools and applications ourselves. If an error occurs in either compilation or during a test flight, the software engineering team will directly edit the source code.

We will only spend 20 minutes doing Bash and Python scripting for today's workshop. The reason is that all the new members coming into this club have different levels of experience and skill sets. We do not want to bore the members who have done Bash and Python scripting before. Having said that, for members who want greater in-depth knowledge of such concepts, we will always provide guides that you can read during your free time. During free time, you are welcome to approach any of the returning members for further assistance in Bash/Python scripting.

If you have not had extensive experience using Git control, we expect you to learn Git during your free time.

Setup

Hopefully you would have partially completed the software setup guide. For Windows users, make sure to complete up to step 8. You will complete the rest of the setup guide in your free time. You will be unable to use Docker for now.

Clone the GitHub repository. Since the drone_code is a private repository, you will be denied permission to clone the repository unless you setup the ssh-agent.

    git clone git@github.com:uas-at-ucla/drone_code.git

Please note that you must have the entire software setup completed before Thursday's web development and communications workshop. For those who wish to develop on Windows, it will take some time to complete the setup guide.

Windows Limitation

  • It is highly doubtful that controls would work on Windows. Our leads have tested controls with no success, and do not rank it as a high priority to add support for this platform. If you are interested in controls development and have a Windows machine, then consider dual-booting Ubuntu 18.04 or using a Virtual Machine.
  • Ground station development is guaranteed to work on Windows.
  • Vision development is guaranteed to work on Windows.

Bash Commands

For Bash explanations, use the explain shell website. If you do not know a command listed in the below examples, enter the commands in the explain shell website. Press the TAB button to autocomplete file names, directories, or shell commands.

Example 1

    echo $USER
    cd ~
    mv drone_code uas
    ls uas
    cd uas/
    ls
    cd src/ground/
    grep -rnwi todo

Example 2

    cd .. && cd ..
    cd tools/scripts
    cat install.sh
    cat install.sh | grep nodejs
    sudo apt install python2.7 python3
    ./install.sh
    which nodejs

Example 3

    cd ~/
    mkdir junk
    cd junk
    touch junk.sh
    nano junk.sh
    chmod +x junk.sh
    ./junk.sh

You will use Example 3 for Bash scripting and similarly for Python scripting in this workshop.

Bash Scripting

Bash scripting is essentially executing a list of Bash commands.

Example 1

#!/bin/bash

function check_if_installed {
  which $1 > /dev/null 2>&1
  echo "Packages all installed."
  return $?
}

function install_package {
  check_if_installed "$1"
  if [ $? -ne 0 ]
  then
    if [ "$PLATFORM" == "Darwin" ]
    then
      brew install $1
    else
      sudo apt install -y $1
    fi
  fi
}

install_package "tmux git nodejs"

Python Scripting

Python is a really easy programming language to learn, especially after you complete CS 31. If you do not know how to do something in Python, then either do a Google search or check Stack Overflow. Other than the examples, the only other stuff you would Google search for Python involves object oriented programming, using new libraries, and perhaps pip installations.

Copy the below python examples one at a time. To paste either in the shell or in nano, press Ctrl+Shift+v. After you save the file, run python FILE_NAME in the command line.

Example 1

print 'Hello world!'
friends = [ "ivan", "ryan", "ben" ]

for names in friends:
    print names

def greet(person):
    print 'Salutations', person

greet("comran")

Example 2

#!/bin/python3

import Queue
import threading
import urllib2

# called by each thread
def get_url(q, url):
    q.put(urllib2.urlopen(url).read())

theurls = ["https://uasatucla.org/","http://google.com", "http://yahoo.com"]

q = Queue.Queue()

for u in theurls:
    t = threading.Thread(target=get_url, args = (q,u))
    t.daemon = True
    t.start()

s = q.get()
print s

In vision, you will use multithreading. In Example 2, each subthread is waiting for a URL to resolve and to respond in order put its content into the queue. Each thread is running on a daemon, which means it will not keep a process up if the main thread ends.

The main thread does a get on the queue to wait until one of the urls does a put. The main thread then emits the first url and terminates. Queues are the best way to farm out work to threads and/or collect the work's results because queues maintain a ranking by using the First-In First-Out order.

In ground station development, you will use asynchronous programming. Understanding the concepts of multithreading, context switching, parallel programming, and asynchronous programming will prove to be vital in the software development process.

Example 3

import os

# Change directory to location of the python script
os.chdir(os.path.dirname(os.path.realpath(__file__)))

import subprocess
command = "wget -O spam.html https://uasatucla.org/wiki/"
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()

In Example 3, you are executing a Bash command in a Python script.

Example 4

View the uas.py file. This file requires further development and stability. The Testing and DevOps subteam's very responsibility is to provide an easy interface for the rest of the UAS@UCLA team to use.

    cat /path/to/uas/tools/scripts/uas.py

Using Docker

See the Docker guide for an explanation of what Docker is.

On MacOS, to get docker running and make the docker command accessible in Terminal, run source tools/scripts/docker/start_machine_mac.sh.

Important: Whenever you are done working with UAS/Docker code, run ./uas.sh cleanup_docker and answer "y" to the prompts.

For the above commands, make sure to first run cd path/to/drone_code

Example 1

This is a practical use of running Docker on the command line. Have two separate terminals open. On one terminal, enter the below command.

    docker run -it ubuntu bash

On the second terminal, enter the below commands.

    docker ps
    docker kill CONTAINER_ID

Notice how in the other terminal that the ubuntu bash docker was terminated.

Example 2

FROM python:3.6-slim

# install dependencies
RUN apt-get update
RUN apt-get install -y libglib2.0-0 libsm6 libxrender1 libxext6 libtk8.6 git cython

COPY ./src/vision/build/vision_requirements.txt /vision_requirements.txt
RUN pip install -r /vision_requirements.txt

# build and install darkflow
RUN pip install Cython
RUN git clone https://github.com/thtrieu/darkflow
RUN pip install /darkflow

# copy needed files
COPY ./src/vision /suas/src/vision
COPY ./lib /suas/lib

# expose vision server and ssh
EXPOSE 8099 22

VOLUME ["/suas/src/vision/data_local"]

WORKDIR /suas/src/vision
ENTRYPOINT ["python3", "vision.py"]

This is a Dockerfile made for Vision. All of these Bash commands are done in the Docker container. To see how this Docker container gets build, refer to the docker_build.sh file in the /path/to/uas/tools/scripts/docker/vision/ directory.

The uas.sh Command

The ./uas.sh command is the UAS@UCLA universal interface to build and execute all the source code. The below command displays all the arguments that you can input.

    cd /path/to/uas/
    ./uas.sh

Vision Example

Enter the below commands to setup and run the Vision Docker.

    ./uas.sh vision build
    ./uas.sh vision run server

On a separate terminal, execute the below commands.

    cd /path/to/uas/
    ./uas.sh vision run client --help
    ./uas.sh vision run client yolo

Controls Example

Enter the below commands to setup and run the Controls Docker. Note that there are currently errors in the Windows environment. We suspect the errors stem from permission issues with Bazel.

    ./uas.sh build
    ./uas.sh simulate

On a separete terminal, execute the following command.

    tmux a -t uas_env

Ground Station Example

Errors currently exist and Ryan Nemiroff will take care of them.

    ./uas.sh ground

Jenkins

See the Jenkins guide for an explanation of what Jenkins is. The Jenkins server is constantly checking all pull requests made in the GitHub repository. This is the definition of continuous integration (CI).

Example 1

The below code is the Jenkinsfile in the UAS repository. Please note that it is the Testing and DevOps subteam's job to figure out the software-in-the-loop and hardware-in-the-loop code.

pipeline {
  agent any

  options {
    ansiColor('xterm')
  }

  stages {
    stage('SETUP') {
      steps {
        fileExists './uas.sh'
        sh './uas.sh nuke'
        sh './uas.sh run_env'
      }
    }
    stage('INITIAL CHECKS') {
      parallel {
        stage('ENVIRONMENT INFO') {
          steps {
            sh 'date'
            sh 'env'
            sh 'pwd'
          }
        }
        stage('LINT') {
          steps {
            sh './uas.sh lint --check'
          }
        }
      }
    }
    stage('BUILD') {
      parallel {
        stage('BUILD CONTROLS') {
          steps {
            sh './uas.sh build'
          }
        }
        stage('BUILD VISION') {
          steps {
            sh './uas.sh vision build'
          }
        }
      }
    }
    stage('TEST') {
      parallel {
        stage('SITL TESTS') {
          steps {
            echo 'Test SITL'
          }
        }
        stage('HITL TESTS') {
          steps {
            echo 'test hitl'
          }
        }
        stage('UNIT TESTS') {
          steps {
            echo 'unit tests'
            sh './uas.sh unittest'
          }
        }
      }
    }
    stage('STATIC ANALYZER') {
      steps {
        echo 'Static analyzer'
      }
    }
  }
  environment {
    PATH = "/usr/local/bin:/usr/bin:/bin:$PATH"
  }
  post {
    always {
      sh 'docker kill $(docker ps --filter status=running --format "{{.ID}}" --latest --filter name=uas_env) || true'
    }
  }
}

Software-in-the-Loop

Software in the Loop (SIL) is testing any software/firmware/algorithm/control system in such a way that a piece of software simulating a piece of hardware, or simulating a physical component, or a physical system, including possibly its response or other characteristics, is in your communication stream in a system which is either open-ended (feed-forward only), or with feedback.

Hardware-in-the-loop

Hardware in the Loop (HIL) is testing any software/firmware/algorithm/control system in such a way that a piece of actual hardware is in your communication stream in a system which is either open-ended (feed-forward only), or with feedback.

Jenkins Master and Slave

Note that Master and Slave are software terminologies. This terminology may be changed to server and clients. Refer to this Quora link for the definition of Master and Slave. To view the respective master and slave files, execute the below command.

    cat /path/to/uas/tools/dockerfiles/jenkins_server/Dockerfile
    cat /path/to/uas/tools/dockerfiles/jenkins_slave/Dockerfile

Automate Machine Learning with Jenkins

The purpose of Jenkins is to ease development by automating everything. One of the very first tasks for the Test and DevOps team to complete this Fall Quarter is to automate the machine learning training for the Vision sub-team.

Feedback Form

Please fill out this Google form with your feedback about this workshop. Since this is the first year that UAS@UCLA is allocating hours training new members with workshops, we always want your feedback. The feedback you give will help Leadership figure out how to improve workshops.

Reminders

Air Frame Meeting is this Wednesday (10/10) in the AIAA Lab at 6:00 PM.

General Meeting is this Thursday (10/11) in Engr IV 38-138 at 6:00 PM.

Web Development and Communications Workshop will follow after the general meeting in Engr IV 47-124 at around 6:45 PM.

ACM AI will have a general meeting this Friday (10/12) in the Northwest Campus Auditorium at 6:00 PM. See the Facebook event for more details.

Free Time

Complete the setup guide. You are welcome to explore the code in the repository or socialize with your neighbors. If you are having technical difficulties, ask any of the returning members for help.

Additional Resources

Please review the additional resources in your free time.