Tuesday, August 29, 2017

Docker Java Example Part 3: Transmode Gradle plugin

At last we get to some Docker in this Docker example.

Gradle Docker Plugin

There are a few prominent docker plugins for gradle: transmode, bmuschko, and Netflix nebula. First, I used transmode, as recommended in the spring boot guide Spring Boot with Docker. After adding the Dockerfile:

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ADD target/java-docker-example-0.0.1-SNAPSHOT.jar app.jar
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]

, and updating build.gradle as described in the guide, I was able to build a docker image for my application:

$ gw clean build buildDocker --info

Setting up staging directory.
Creating Dockerfile from file /Users/ryanmckay/projects/java-docker-example/java-docker-example/src/main/docker/Dockerfile.
Determining image tag: ryanmckay/java-docker-example:0.0.1-SNAPSHOT
Using the native docker binary.
Sending build context to Docker daemon  14.43MB
Step 1/5 : FROM openjdk:8-jdk-alpine
---> 478bf389b75b
Step 2/5 : VOLUME /tmp
---> Using cache
---> 136f2d4e58dc
Step 3/5 : ADD target/java-docker-example-0.0.1-SNAPSHOT.jar app.jar
---> b3b47b89bbf1
Removing intermediate container 92f637bc67e0
Step 4/5 : ENV JAVA_OPTS ""
---> Running in e90c9a3557eb
---> 1d3f6526e8e5
Removing intermediate container e90c9a3557eb
Step 5/5 : ENTRYPOINT sh -c java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar
---> Running in 2fbfb52f836d
---> f001bdddc80b
Removing intermediate container 2fbfb52f836d
Successfully built f001bdddc80b
Successfully tagged ryanmckay/java-docker-example:0.0.1-SNAPSHOT

$ docker images
REPOSITORY                       TAG               IMAGE ID        CREATED          SIZE
ryanmckay/java-docker-example    0.0.1-SNAPSHOT    f001bdddc80b    3 minutes ago    115MB

$ docker run -p 8080:8080 -t ryanmckay/java-docker-example:0.0.1-SNAPSHOT 

Automatically Tracking Application Version

Note that the Dockerfile at this point has the application version hard-coded in it. This duplication must not stand. The transmode gradle plugin also supports a dsl for specifying the Dockerfile in build.gradle. Then as part of the build process, it produces the actual Dockerfile.

I set about moving line by line of the Dockerfile into the dsl. With one exception it went smoothly. You can see the result in v0.3 of the app. The relevant portion of build.gradle is listed here. You can see its pretty much a line for line translation of the Dockerfile. And since we have access to the jar filename in the build script, nothing needs to be hard coded for docker.

// for docker
group = 'ryanmckay'

docker {
 baseImage 'openjdk:8-jdk-alpine'
}

task buildDocker(type: Docker, dependsOn: build) {
 applicationName = jar.baseName
 volume('/tmp')
 addFile(jar.archivePath, 'app.jar')
 setEnvironment('JAVA_OPTS', '""')
 entryPoint([ 'sh', '-c', 'java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar' ])
}

ENV is used at build time And at run time

One little gotcha in the previous section lead to an interesting learning. My initial attempt to set the JAVA_OPTS env variable looked like this:
setEnvironment('JAVA_OPTS', '')
but that produced an illegal line in the Dockerfile:
ENV JAVA_OPTS
That led me to read about the ENV directive in the Dockerfile reference docs.  I was confused about whether ENV directives are used at build time, run time, or both.  Turns out, the answer is both, as I was able to prove to myself with the following. The ENV THE_FILE is used at build time to decide which file to add to the image, and at run time as an environment variable, which can be overriden at the command line.

$ cat somefile
somefile contents

$ cat Dockerfile
FROM alpine:latest
ENV THE_FILE="somefile"
ADD $THE_FILE containerizedfile
ENTRYPOINT ["sh", "-c", "cat containerizedfile && echo '-----' && env | sort"]

$ docker build -t envtest .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM alpine:latest
 ---> 7328f6f8b418
Step 2/4 : ENV THE_FILE "somefile"
 ---> Using cache
 ---> 148a4236ce19
Step 3/4 : ADD $THE_FILE containerizedfile
 ---> Using cache
 ---> d44f9e242685
Step 4/4 : ENTRYPOINT sh -c cat containerizedfile && echo '-----' && env | sort
 ---> Running in 70de8ceac5ef
 ---> 5d875712904a
Removing intermediate container 70de8ceac5ef
Successfully built 5d875712904a
Successfully tagged envtest:latest

$ docker run envtest
somefile contents
-----
HOME=/root
HOSTNAME=36b3233697df
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
THE_FILE=somefile
no_proxy=*.local, 169.254/16

$ docker run -e THE_FILE=blah envtest
somefile contents
-----
HOME=/root
HOSTNAME=6a0f8c183a18
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
THE_FILE=blah
no_proxy=*.local, 169.254/16

Version Tag and Latest Tag

So that all worked fine for creating and publishing a versioned docker image locally.  But I really want that image tagged with the semantic version and also the latest tag.  The transmode plugin currently does not support multiple tags for the produced docker image.  I'm not the only one who wants this feature.  I took a look at the source code, and it wouldn't be a minor change.  At this point, I'm only publishing locally, so given the choice between version tag and latest tag, I'm going to go for latest for now.  This is a simple matter of adding tagVersion = 'latest' to the buildDocker task.

I tagged the code repo at v0.3.2 at this point.

I'm going to move on to evaluating the bmuschko and Netflix Nebula Docker Gradle plugins next.

Wednesday, August 23, 2017

Docker Java Example Part 2: Spring Web MVC Testing

The next step was to add some tests. The tests that came with the demo controller used a Spring feature I was not familiar with, MockMvc. The Spring Guide "Testing the Web Layer" provides a good discussion of various levels of testing, focusing on how much of the Spring context to load. There are 3 main levels: 1) start the full Tomcat server with full Spring context, 2) full Spring context without server, and 3) narrower MVC-focused context without server. I wanted to compare all three, plus add in variation in testing framework and assertion framework. Specifically I wanted to add Spock with groovy power assert.  The aspects I wanted to compare were: test speed, readability of test code, readability of test output.  I intentionally made one of the tests fail in each approach to compare output.


Spock with Full Tomcat Server

This is the approach I am most familiar with.
https://github.com/ryanmckaytx/java-docker-example/blob/v0.2/src/test/groovy/net/ryanmckay/demo/GreetingControllerSpec.groovy

Timing

I ran and timed the test in isolation with
$ ./gradlew test --tests '*GreetingControllerSpec' --profile

Total 'test' task time (reported by gradle profile output): 13.734s
Total test run time (reported by junit test output): 12.690s
Time to start GreetingControllerSpec (load full context and start tomcat): 12.157s
So, not fast. Maybe one of the other approaches can do better.

Test Code Readability

def "no Param greeting should return default message"() {

    when:
    ResponseEntity<Greeting> responseGreeting = restTemplate
                .getForEntity("http://localhost:" + port + "/greeting", Greeting.class)

    then:
    responseGreeting.statusCode == HttpStatus.OK
    responseGreeting.body.content == "blah"
}
I really like Spock. I like the plain English test names. I like the separate sections for given, when, then, etc. I think it reads well and makes it obvious what is under test.

Test Output Readability

When a test fails, you want to see why, right?  In this aspect, groovy power assertions are simply unparalleled.
Condition not satisfied:

responseGreeting.body.content == "blah"
|                |    |       |
|                |    |       false
|                |    |       12 differences (7% similarity)
|                |    |       (He)l(lo, World!)
|                |    |       (b-)l(ah--------)
|                |    Hello, World!
|                Greeting(id=1, content=Hello, World!)
<200 OK,Greeting(id=1, content=Hello, World!),{Content-Type=[application/json;charset=UTF-8], Transfer-Encoding=[chunked], Date=[Tue, 22 Aug 2017 22:06:33 GMT]}>

 at net.ryanmckay.demo.GreetingControllerSpec.no Param greeting should return default message(GreetingControllerSpec.groovy:27)
Note that the nice output for responseGreeting itself comes from ResponseEntity.toString(), and from Greeting.toString(), which is provided by Lombok.

Spock with MockMvc

By adding @AutoConfigureMockMvc to your test class, you can inject a MockMvc instance, which facilitates making calls directly to Springs HTTP request handling layer.  This allows you to skip starting up a Tomcat server, so should save some time and/or memory.  On the other hand, you are testing less of the round trip, so the time savings would need to be significant to justify this approach.
https://github.com/ryanmckaytx/java-docker-example/blob/v0.2/src/test/groovy/net/ryanmckay/demo/GreetingControllerMockMvcSpec.groovy

Timing

This approach was about 500ms faster than with tomcat.  Not significant enough to justify for me, considering the overall time scale.

Total 'test' task time (reported by gradle profile output): 13.263s
Total test run time (reported by junit test output): 12.281s
Time to start GreetingControllerSpec (load full context, no tomcat): 11.804s

Test Code Readability

def "no Param greeting should return default message"() {

    when:
    def resultActions = mockMvc.perform(get("/greeting")).andDo(print())

    then:
    resultActions
            .andExpect(status().isOk())
            .andExpect(jsonPath('$.content').value("blah"))
}
This reads reasonably well.  Capturing the resultActions in the when block to use later in the then block is a little awkward, but not too bad.  Being able to express arbitrary JSON path expectations is convenient.  I didn't see an obvious way to get a ResponseEntity as was done in the full Tomcat example.

Test Output Readability

Condition failed with Exception:

resultActions .andExpect(status().isOk()) .andExpect(jsonPath('$.content').value("blah"))
|              |         |        |        |         |                     |
|              |         |        |        |         |                     org.springframework.test.web.servlet.result.JsonPathResultMatchers$2@1f977413
|              |         |        |        |         org.springframework.test.web.servlet.result.JsonPathResultMatchers@6cd50e89
|              |         |        |        java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!&rt;
|              |         |        org.springframework.test.web.servlet.result.StatusResultMatchers$10@660dd332
|              |         org.springframework.test.web.servlet.result.StatusResultMatchers@251379e8
|              org.springframework.test.web.servlet.MockMvc$1@68837646
org.springframework.test.web.servlet.MockMvc$1@68837646

 at net.ryanmckay.demo.GreetingControllerMockMvcSpec.no Param greeting should return default message(GreetingControllerMockMvcSpec.groovy:29)
Caused by: java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!&rt;

This test output does not read well at all. Spock and the Spring MockMvc library are both tripping over each other trying to provide verbose output.  I think you need choose either Spock or MockMvc, but not both.

JUnit with WebMvcTest and MockMvc

This configuration is on the far other end of the spectrum from full service Spock.  With @WebMvcTest, not only does it not start a Tomcat server, it doesn't even load a full context.  In the current state of the project this doesn't make much of a difference because the GreetingController has no injected dependencies.  If it did, I would have to mock those out.  Again, because of the differences from "real" configuration, time savings would need to be significant.
https://github.com/ryanmckaytx/java-docker-example/blob/v0.2/src/test/groovy/net/ryanmckay/demo/GreetingControllerTests.java

Timing

This approach was also about 500ms faster overall than full context with Tomcat.

Total 'test' task time (reported by gradle profile output): 13.275s
Total test run time (reported by junit test output): 0.269s
Time to start GreetingControllerSpec (load narrow context, no tomcat): 11.88s

Test Code Readability

@Testpublic void noParamGreetingShouldReturnDefaultMessage() throws Exception {

    this.mockMvc.perform(get("/greeting")).andDo(print()).andExpect(status().isOk())
            .andExpect(jsonPath("$.content").value("blah"));
}

This is the least readable for me.  Again, I like separating the call under test from the assertions.

Test Output Readability

The failure message for MockMvc-based assertion failures isn't as informative as Spock in this case.
java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!>" type="java.lang.AssertionError">java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!>

Because the test called .andDo(print()), some additional information is available in the standard out of the test, including the full response status code and body.

Conclusion

I'm as convinced as ever that Spock is the premier Java testing framework.  I'm reserving judgment on the Spring annotations that let you avoid starting a Tomcat server or load the full context.  If the project gets more complicated, those could potentially provide a nice speedup.

I tagged the code repo at v0.2 at this point.

Sunday, July 23, 2017

Docker Java Example Part 1: Initializing a new Spring Boot Project

I've been wanting to learn more about Docker for a while.  I'm almost done with this udemy course Docker for Java Developers.  Its a good course, and the concepts are straightforward.  To help me commit it to memory, I wanted to do my own project to apply what I'm learning.
https://github.com/ryanmckaytx/java-docker-example
In this part, I'm just going to initialize a new project.  Part 2 covers Spring Web MVC testing.

Set up your dev machine

Every once in a while, like when you switch jobs and get a new laptop, you need to set up a machine for java development.  

For this, I like to use sdkman.  It helps install the tools you need to do java development.  It also helps switch between multiple versions of those tools.  I've installed java, groovy, gradle, and maven.
$ sdk c

Using:

gradle: 4.0
groovy: 2.4.11
java: 8u131-zulu
maven: 3.5.0

Create a new project

There are a few good ways to do this.  The typical way I do this is to copy another project.  Within an organization, or at least within a team, there is typically some amount of infrastructure and institutional knowledge built into existing projects that you want in a new project.  But for this project, I wanted to practice starting completely from scratch.  There are a couple of good options.

Gradle init

I like gradle as a build tool, and gradle has a built in project initializer.  It supports a few project archtypes, including pom (converting a maven project to gradle), java library, and java application.  It even explicitly supports spock testing framework.

$ gradle init --type java-application --test-framework spock

BUILD SUCCESSFUL in 0s
2 actionable tasks: 2 executed
$ tree
.
|____build.gradle
|____gradle
| |____wrapper
| | |____gradle-wrapper.jar
| | |____gradle-wrapper.properties
|____gradlew
|____gradlew.bat
|____settings.gradle
|____src
| |____main
| | |____java
| | | |____App.java
| |____test
| | |____groovy
| | | |____AppTest.groovy

You can see it also generates a demo App and AppTest.

Spring Boot Initializr

For Spring Boot apps, Spring provides the Spring Boot Initializr.  This lets you choose from a curated  (but extensive) set of options and dependencies, and then generates a project in a zip file for download.  Similarly to gradle init, it includes a default basic app and test.



$ unzip demo.zip
Archive:  demo.zip
   creating: demo/
  inflating: demo/gradlew
   creating: demo/gradle/
   creating: demo/gradle/wrapper/
   creating: demo/src/
   creating: demo/src/main/
   creating: demo/src/main/java/
   creating: demo/src/main/java/net/
   creating: demo/src/main/java/net/ryanmckay/
   creating: demo/src/main/java/net/ryanmckay/demo/
   creating: demo/src/main/resources/
   creating: demo/src/main/resources/static/
   creating: demo/src/main/resources/templates/
   creating: demo/src/test/
   creating: demo/src/test/java/
   creating: demo/src/test/java/net/
   creating: demo/src/test/java/net/ryanmckay/
   creating: demo/src/test/java/net/ryanmckay/demo/
  inflating: demo/.gitignore
  inflating: demo/build.gradle
  inflating: demo/gradle/wrapper/gradle-wrapper.jar
  inflating: demo/gradle/wrapper/gradle-wrapper.properties
  inflating: demo/gradlew.bat
  inflating: demo/src/main/java/net/ryanmckay/demo/DemoApplication.java
  inflating: demo/src/main/resources/application.properties
  inflating: demo/src/test/java/net/ryanmckay/demo/DemoApplicationTests.java

Pretty much the only thing I don't like here is that the test isn't spock and there doesn't seem to be a way to choose it. Not a big deal, its easy to change afterward.  I went with initializr for this project.

JHipster

JHipster is an opinionated full-stack project generator for Spring Boot + Angular apps.  It has a lot of features that I want to explore later, so for now I stuck with Spring Boot Initializr.

Add a Rest Controller

I could have gone straight to CI at this point, but I wanted to do a little extra work that I wish the Initializr could have done for me.  I copied in a basic hello world spring rest controller (and its tests) from the Spring Restful Web Service Guide.  The repo is now at this point https://github.com/ryanmckaytx/java-docker-example/tree/v0.1

Saturday, June 24, 2017

Interview Prep/Tips


I recently interviewed for a new job, which led me to review some notes I made for myself after some interviews a couple years ago.  I thought I would share them in case someone else might find something useful here.  The point of preparing is not to pretend to be anything you aren't or to know anything you don't, the point is just to have everything you do know on the tip of your tongue.  You only have probably an hour per panel, so you need to be on point.

Make a list of professional things that are important to you.  Out of those, pick the most important and write a few sentences.  Those are kind of your mission statement.  I put those sentences right at the top of my resume.  Here is my list:
  • Working with smart, motivated people
  • Agile
  • Mentoring
  • Being Mentored
  • DevOps
  • Physical fitness
  • Feeling like I'm contributing to company's success
  • Company mission
  • Competence of other departments
  • Trust in leadership
  • Trust from leadership
  • Advanced software architecture
  • Advanced technology
  • Work/Life balance
Make a list of the major lessons learned during your time at your current job.  These can include accomplishments, things you wish you had done better, or just experience gained.

Make a list of what areas you want to learn more/grow more in.  These should be things that really excite you.  It can be stuff you already do in your current job or not.  Its helpful if you've shown some initiative and done some learning on your own outside of work.

Make a list of recent videos watched, books read, conferences attended, and one or two sentences about each.  Try to tie the things you learned in these back to the items in the other lists. You want to show that you are passionate about your craft and always learning and improving.

Learn about the prospective company, their products, their market, their competition. Make a list of questions you have about the prospective company.  Some of these should tie back to what you are passionate about.  For me it was a lot of questions about how they do agile.  What the dev teams look like.  How they interact with other departments like product and sys ops.  Questions show you are interested, passionate, and evens out the power structure of the interview a bit, which makes everybody feel more comfortable.  Yes, they are interviewing you, but you are also interviewing them.

For design problems, stay calm and focus on fundamentals.  Treat the interviewer as a subject matter expert/product owner.  Don’t be afraid to ask questions!  Your job is to:
  • Capture the ubiquitous language of the domain.  Make sure you understand what the pieces are and what they do. 
  • Capture the functional requirements of the application.  I like to focus on user stories/use cases.  Just solve one at a time, and evolve the design to handle additional ones.
  • Nouns you hear are good candidates for objects/resources.  Verbs are good candidates for methods.
  • Keep it lean.  Start simple and try to deliver a minimal top to bottom slice that does something.  Then move on to more complicated scenarios iteratively.

      Wednesday, May 24, 2017

      Stop Over-Engineering

      I just watched Greg Young's Build Stuff 2016 Keynote - Stop Over-Engineering, and had a lot of good takeaways.

      Your software is only part of an overall business process system.  You don't have to solve every problem using software.  Definitely not early on, and typically not ever.  Why wouldn't you try to solve all the problems?  Because the cost benefit tradeoff doesn't make sense.  Many tasks are less expensive for humans to do than to try to automate.  Instead of trying to handle every edge case, regardless of how infrequently it might be encountered, focus on handling the happy path, and detecting when the user has gone off the happy path.  When that happens, hand it off to humans.

      How do you know what is the happy path?  He used the analogy, "Stop watering the weeds in your life and start watering the flowers".  How do you know where the flowers are? Data.

      This is where a distinction was made between brown-field projects and green-field projects.  Brown-field projects have the advantage of usage data.  You can see which features are used the most and how they are used.  So it is easer to make data-driven decisions about where to invest effort.

      He gave an example of an invoicing app.  If you try to capture all the requirements for an invoicing system, you will be in endless meetings.  This struck a chord with me, because in a past job, I worked on a brownfield project in the consumer financial sector, and I had exactly this same experience.  In Greg's case, after 2 weeks of meetings, they decided to stop capturing exhaustive requirements, and instead, look at the past year's worth of actual invoices.  Then the domain experts were simply asked to classify the invoices wrt whether they were on the happy path or not, and how to detect getting outside the happy path.  In one day, they were able to implement a solution that solved 60-70% of the previous year's invoices.  Then they looked at the next most common case (15%) and solved that the next day.  And the next (6%).  After 2 weeks they got to 99%, and then they stopped.  They had reached the point of diminishing returns on investment.  That project had been budgeted to take 9 months.  Don't spend 4 days of modeling to automate a task that takes a human 5 minutes of work once a year!

      The problem with green-field projects is that there is no data.  He asks the question, how many of us have built features that have never been used?  How many of us have build whole products that have never been used?  So how do we know where the flowers are?  We need to get data.  How do we get data?  Greg described two complementary approaches: Throw sh*t at the wall (and see what sticks), and human concierge service.  The first approach is to build small, unpolished pieces of functionality and see what people actually like.  Taken to the extreme, you get the Feature Fake, where you make it look like you have added a new feature to your application, and see who shows interest in it.

      The human concierge service actually has no automation at all.  Humans do all the work for a while, so that you can gather usage data, and find the happy path to automate first.  Human concierge is one of several lean experimentation techniques.

      Some other bullet points:

      • Most things in software aren't worth building
      • Cost Benefit Analyze everything!  Where is your break even point?  If you can't figure it out - stop.  And figure it out.
      • Every developer should hire another developer to build something, to see the CBA from the other side.

      Saturday, August 1, 2015

      Agile Peer Review

      Preface

      What follows is primarily the result of nearly 2 years of intentional refinement by an experienced, high-performing agile team. Dev team members, architects, product owner, and even members of other dev teams participated and helped shape our thoughts on review. I also incorporated ideas from several other experienced peers. But like anything agile, it's a work in progress.

      Don't let preconceptions about the term "review" color your reading - most of the reviews called for are only 5-15 minutes long.

      Overview

      Functional defects and design problems get exponentially more expensive the longer they are allowed to exist. The earlier those problems can be found and fixed, the better. Peer reviews provide an inexpensive, effective method for doing that.

      There are several major work products in the lifecycle of a user story that warrant review (these are discussed in detail below):
      • Domain Model - The team’s shared, fundamental, documented understanding of the domain.
      • Demo script - How the dev team will demonstrate to stakeholders that the story is functionally complete.
      • Design - Focused on the API and its test cases, but also covering important implementation details. 
      • Pull Request - A single coherent change to the shared codebase, in service of a story. 
      • Code Complete - The full set of changes to the codebase that were made to complete the story.

      Benefits

      These reviews deliver tangible value to the business

      • Higher-quality code 
      • Higher, more consistent velocity

      And additional value to the dev team

      • Pride in work 
      • Self-improvement

      As a result of the direct effects of the reviews

      • Getting a better product, sooner 
        • Bugs identified, eliminated earlier in the process 
        • More maintainable code
        • Better tested
        • More adherent to team/organizational standards
      • Knowledge sharing
        • Team members sharing the same frame of mind
        • Better understanding of the domain
        • Better understanding of how system components interact
        • Improving our craftsmanship every day by learning from each others strengths
        • Better understanding of our tools and technical stack

      What to Review

      Domain Model

      The fundamental, documented understanding of the domain shared among the whole team including the Product Owner. A critical part of this model is the Ubiquitous Language, which unambiguously expresses domain concepts, enabling effective communication. Any change in that understanding or expression needs to be reviewed by the rest of the team, to keep everyone on the same page, and to consider potential impact to the software design. Not every story changes the Domain Model.

      Demo Script

      How the dev team will demonstrate to stakeholders that the story is functionally complete. This would involve the dev team and Product Owner. Depending on team/org structure, more or less of this might be provided to the dev team by PO, or might even be developed collaboratively, in which case a separate review would be unnecessary. 

      There is not a lot of detail here, just a handful of user-facing scenarios. If it is more than a handful, the story is probably too big.. It shouldn't take much time at all to produce or review. If it does take a long time, again it's probably too big, or there is a lack of understanding that needs to be addressed before proceeding. For example:
      • Create an event
        • Go to the Event list page
        • Click "New Event" button
        • Fill in information in Event form, click save
        • Should show Event overview page with saved info
        • Go back to Event list page, new event should be there
      • Register for an Event
        • Registrant Role
          • Go to Event Registration page Fill in form, submit
        • Admin Role
          • Go to Event overview page, see new Registration

      Design

      For a microservice, this would focus on the API and its test cases, but would also cover important implementation details. Generally should be developed collaboratively with the major client(s) of the API, e.g. UI. In which case, the review with the rest of the team shouldn't take long.
      • Resource endpoints
      • Methods on those endpoints
      • API objects
      • API test cases
      • API versioning
      • New runtime dependencies, e.g. now we're going to have to use the Org service
      • Major changes to infrastructure, e.g. using a new database, new message queue
      • Data migration

      Pair Programming

      This is effectively continuous code review between two team members while the code is being written. I wrote up most of my thoughts on this practice here. Because of the normalizing and quality control forces exerted here, the pull request and code complete reviews go faster.

      Pull Request

      Code gets merged into the shared repository by issueing a pull request. This should occur frequently, typically multiple times per day per developer. It should be coherent and correct, but not necessarily complete. Each pull request gets reviewed by one or more other team members. In my experience, the types of things caught here are team standards violations, code/test readability (e.g. method and variable names), and code factoring.

      Code Complete

      When the dev(s) working on a story consider it code complete, the code and its tests should be reviewed by the rest of the dev team, typically plus an architect.
      • The person(s) who did the coding walk everyone else through it
        • Generally start with brief review of the story
        • Then walk through the important parts of the code
          • And the tests!
      • While those persons are presenting, someone else is taking action item notes
        • Somewhere public, or just email out after
      • The story is not considered dev-complete until those action items are addressed.
        • Creating a tech debt story to do it later is a valid option but should not be abused 
      • This is at a level similar to Design review, focusing on API classes and tests, and any deviations from the original design.

      How to Review

      When to review?

      Each work product should be reviewed as early as feasible, because course correction is cheaper the earlier you do it. Also, the less time elapses between creation and review, the more of the context the creator(s) will still have loaded. Finally, by spreading the reviews out over the lifetime of the story (as opposed to reviewing everything at the end), you avoid the tendency to get mentally overwhelmed by the volume of work to be reviewed and just rubber-stamp it. 

      On my team, we had one-week sprints, Monday to Friday. We took Monday morning to commit to a set of stories for the sprint, produce and review demo plans for all the committed stories, and produce and review design and test plan for the first one or two stories we were starting work on. All other reviews were performed as needed at the end of each day.

      Why not just create together instead of reviewing later?

      There is a tradeoff to be made between direct collaboration and ex post facto review. In my experience, you typically want to keep creative activities for a given story limited to 2-3 people, 4 at the most. Otherwise the cost/benefit starts to deteriorate.

      How much time to invest?

      For a team that has been together a while, the amount of time to target for each review is ~30 minutes for code complete review, and 5-15 minutes for the others. Expect more review time early on in a team's formation. However, team norming will drive the time down significantly, and performing these reviews accelerates team norming. You don't keep finding the same things in review, because you incorporate the feedback and get better.

      Who does the reviewing?

      The composition of the reviewing group varies based on what is being reviewed, and larger review groups will tend to use more person-hours, because:
      • There are more people spending the time
      • More people talk more
      • The members of large groups will be less familiar with what is being reviewed

      Putting it all Together: A Sprint in the Life of a Story

      • Monday morning - Dev team meets to commit to stories for sprint, and devise demo script for those stories. Right after that meeting, they meet with Product Owner to review the demo script, adjust as necessary.
      • Monday afternoon - One pair of developers start on a medium-sized story. They come up with a design together. During the design, they realize that their Domain Model and Ubiquitous Language will need to change. Previously, Constituents could only register for Events. Now a Constituent can be designated as the "Event Organizer". They ask the rest of the team and an architect for a design review, scheduled for 30 minutes later. In the mean time, they check the proposed new terminology with the Product Owner, and add the term to the project's wiki. Then they go ahead and start implementing the design.
        In the design review, an overlooked test case is identified and added, and the format of a timestamp field is changed to suit the UI. Minimal rework in the codebase is required. They also take that opportunity to inform the rest of the dev team and the architect of the new Ubiquitous Language term. Monday end of day - The pair issues a pull request. One other developer is still in the office, spends 5 minutes reviewing it. Catches a few issues, which he comments on, but nothing serious, so he merges the pull request.
      • Tuesday, Wednesday - Several more pull requests issued. Since the rest of the team is there, they both review both pull requests. The architect notices the pull requests going by and adds a few comments as well. Late Wednesday afternoon, the pair working on the story determines that it is code complete, and schedules a team code review for end of day.
      • Wednesday end of day - Dev team + Architect code review for the story. One of the dev pair refreshes everyone's memory about the story, then walks through the code and tests. The other member of the dev pair is taking notes. A few minor issues are caught, and one potentially serious scalability issue. The team decides to consult historical metrics to see if it will be a problem short- or long-term. Once they determine that it would take about a year of usage at current rates to become a problem, they decide to add monitoring for this story, and add a tech debt story to address the scaling longer-term.
      • Thursday morning - Dev pair demos the story to the Product Owner according to the demo plan.

      Further Reading

      Sunday, March 29, 2015

      More Vagrant Learnings

      Last time I wrote about how we use Vagrant to manage our local development environment at work. Since then I've dug into Vagrant a little more deeply and found a few nuggets I wanted to share.

      Package your own Vagrant Box

      There are some great base boxes (images) available for vagrant. It will even install them for you. If you want to tweak the box, you can use one or more of the many available provisioning providers. However, sometimes you might want to just bake the tweak into a new base image, for example, if the tweak is something you want to apply to all your instances, or if it is time consuming. Vagrant makes it very easy to do this.

      Initialize, start VM

      $ vagrant init hashicorp/precise32
      $ vagrant up
      

      Install new stuff in VM

      $ vagrant ssh
      vagrant@precise32:~$ echo 'set editing-mode vi' > ~/.inputrc
      vagrant@precise32:~$ sudo apt-get update
      vagrant@precise32:~$ sudo apt-get install vim
      vagrant@precise32:~$ exit
      

      Package up VM image as a new Vagrant box

      $ vagrant package
      ==> default: Attempting graceful shutdown of VM...
      ==> default: Clearing any previously set forwarded ports...
      ==> default: Exporting VM...
      ==> default: Compressing package to: /Users/ryanmckay/projects/vagrant/package.box
      
      $ ls -l
      total 801784
      -rw-r--r--  1 ryanmckay  staff       3001 Mar 14 22:13 Vagrantfile
      -rw-r--r--  1 ryanmckay  staff  410509096 Mar 15 20:32 package.box
      
      $ vagrant box list
      hashicorp/precise32 (virtualbox, 1.0.0)
      
      $ vagrant box add --name 'ryanmckay/withvim' package.box 
      ==> box: Adding box 'ryanmckay/withvim' (v0) for provider: 
          box: Downloading: file:///Users/ryanmckay/projects/vagrant/package.box
      ==> box: Successfully added box 'ryanmckay/withvim' (v0) for 'virtualbox'!
      
      $ vagrant box list
      hashicorp/precise32 (virtualbox, 1.0.0)
      ryanmckay/withvim   (virtualbox, 0)
      

      Provisioning

      Vagrant provides a boatload of provisioning providers, e.g. puppet, chef, ansible, and even shell script. Normally configured provisioners are run when bringing up the VM for the first time. However, you can also run the provisioners with vagrant provision. For example, if you have the following:

      provision_counter.sh shell script

      echo "provisioning"
      if [ -f /etc/provision_count ]; then
       counter=`cat /etc/provision_count`
      else
       counter=0
      fi
      counter=$((counter+1))
      echo -n $counter > /etc/provision_count
      echo "provisioned: $counter"
      

      Configured in your Vagrantfile

      config.vm.provision "shell", path: "provision_counter.sh"
      

      When you vagrant up, provisioners are run

      $ vagrant up
      ...
      ==> default: Running provisioner: shell...
          default: Running: /var/folders/k5/6mc1_m_n1675572ts12qr7588mtjh5/T/vagrant-shell20150329-86419-qkjn7x.sh
      ==> default: stdin: is not a tty
      ==> default: provisioning
      ==> default: provisioned: 1
      

      And you can run provisioners explicitly

      $ vagrant provision
      ==> default: Running provisioner: shell...
          default: Running: /var/folders/k5/6mc1_m_n1675572ts12qr7588mtjh5/T/vagrant-shell20150316-31070-d35qp6.sh
      ==> default: stdin: is not a tty
      ==> default: provisioning
      ==> default: provisioned: 2
      

      You can even add/modify provisioning lines in the Vagrantfile of a running VM, and when you run 'vagrant provision', the new lines will be used.

      Reloading Vagrantfile

      The Vagrantfile is the main config file for your vagrant VM. It specifies all the stuff that needs to be set up before the OS is running, as well as the hooks for provisioning once it has booted. The Vagrantfile can be reloaded by 'vagrant reload', which is basically 'halt' and then 'up'. It does not re-run provisioners unless you tell it to. Some examples of why you would want to do this are to modify your network setup or your shared folders. For example, if you want to:

      Add a network port forward and a shared folder

      config.vm.network "forwarded_port", guest: 80, host: 8080
      config.vm.synced_folder "src/", "/srv/website"
      

      Then 'vagrant reload' would restart your VM with the new configuration without having to reprovision.

      Vagrant SSH

      'vagrant ssh' is how you ssh into an interactive session on your VM. You can also use 'vagrant ssh -c command' to run a command in the VM instead of an interactive shell, similar to 'ssh host command'.
      $ vagrant ssh -c 'hostname'
      precise32
      

      In order to connect to your VM with regular ssh (or anything that depends on ssh, e.g. scp or rsync over ssh), you need to

      Export vagrant ssh-config

      $ vagrant ssh-config
      Host default
        HostName 127.0.0.1
        User vagrant
        Port 2222
        UserKnownHostsFile /dev/null
        StrictHostKeyChecking no
        PasswordAuthentication no
        IdentityFile /Users/ryan.mckay/.vagrant.d/insecure_private_key
        IdentitiesOnly yes
        LogLevel FATAL
      
      $ vagrant ssh-config > ssh-config
      

      Then provide ssh-config to ssh and friends

      $ ssh -F ssh-config default 'hostname'
      precise32
      
      $ touch scp_me
      $ scp -F ssh-config scp_me default:/tmp
      scp_me                              100%    0     0.0KB/s   00:00    
      $ ssh -F ssh-config default 'ls /tmp'
      scp_me
      
      $ mkdir rsync_test
      $ touch rsync_test/rsync_me
      $ rsync -a -e 'ssh -F ssh-config' rsync_test default:/tmp/
      $ ssh -F ssh-config default 'find /tmp'
      /tmp
      /tmp/rsync_test
      /tmp/rsync_test/rsync_me
      /tmp/scp_me