Daily Archives: June 26, 2019

[QCon 2019] Low Latency in the Cloud, with OSS

Posted on by
Reply

Mark Price @epickrram

For other QCon blog posts, see QCon live blog table of contents

Requirements

  • Trading app
  • Need microsecond (not millisecond) response time
  • Need data in memory vs database
  • Lock free programming
  • Redundancy
  • High volume
  • Predictable latency

Hydra

  • System built on OSS
  • Opinionated framework to accelerate app dev
  • Clients communicate with stateless, scalable gateways
  • Persistors – manage data in memory.
  • Gateway – converts large text message to something smaller and more efficient

Design choices

  • Replay logs to reapply changes. Business logic must be fully deterministic. Bounded recovery times
  • Placement group in cloud – machines guaranteed to be near each other. Minimizes latency between nodes

Testing latency

  • Do as part of CD pipeline
  • Can’t physically monitor with fibertab
  • Capture in histogram to get statistical view and calculate data
  • Test under load
  • Fan out where test from
  • Store % in time series data
  • Can see jigger for garbage collection

Performance on shared box/cloud

  • Not in control of resources running on
  • Containers share L3 cache so can see higher rates of cache miss
  • CPU throttling effects
  • Hard to measure since can’t see what neighbors are doing
  • One option is to rent the largest box possible and compare to vendor website for specs. If have max # cores, know have box to self. Expensive. Was about five dollars a year. At that price, might be worth just buying own machine in data center
  • Can pack non latency services onto shared machines

<missed some at the end. I got an email that distracted me>

My impressions

There was a lot of discussion about the histogram. I would have liked to see some examples rather than just talking about how it is calculated. They didn’t have to be real examples to be useful. There were some interesting facts and it was a good case study so I’m glad I went. I was glad he addressed that non-cloud is a possible option for this scenario

[QCon 2019] Making npm install safe

Posted on by
Reply

Kate SIlls @kate_sills

For other QCon blog posts, see QCon live blog table of contents

General

  • Building financial software in JavaScript
  • 97% of code in a modern web app comes from npm

Security issues

  • All packages are risky
  • Imports and global variables
  • Effects opaque
  • Can be from dependency many levels deep

Pattern

  • Event stream package (11/28/18)
  • Electron native notify package (6/4/19)
  • Can call node built it modules to read a file and send it over the network
  • Targetted cryptocurrency

Options for solution

  • Write everything yourself – not scalable
  • Pay open source maintainers so someone responsible for security – people make mistakes. Even people who are paid can compromise a system
  • Code audits – don’t see everything, Hard to find clever 

Other approach

  • Preventing attacks requires infallability
  • Better to look for ways to limit damage
  • For example, would be better of if can’t import fs
  • JavaScript is good at code isolation. Clear separation between pure computation and connection to outside world

Realms – draft proposal

  • Want to be able to create realm without overhead of an iframe
  • Featherweight compartment – shares primordials/context
  • There is a realm shim now
  • Self/window not defined in the compartment

Attack – prototype poisoning

  • Save copy of original function
  • Do something bad first and then call original function so it looks right
  • SES (Secure ECMAScript)  – realms + transitive freezing/hardening
  • Can’t change prototype behavior with SES
  • npm install ses
  • SES.makeSESRootRealm()

POLA

  • Principle of least authority
  • Same as principle of least privilege
  • Reasonable to want to access file system. Can attenuate (reduce the impact of) access by wrapping fs with check for correct file name. (Not clear how prevents using original fs). Method harden protects
  • The chalk package needs process/OS access to change color
  • But can kill process and change priority of process with that access
  • Want to limit access to just what needed
  • Chalk only needs OS to get the release. Can attenuate so just have that one function to return release string.
  • Object capabilities – http://habitatchronicles.com/2017/05/what-are-capabilities/

Moddable XS

  • Only completed ECMA Script 2018 engine optimized for embedded device
  • Contains SES
  • Safe for users to install JS apps
  • Can only do specific things
  • Can add own app to washing machine

Metamask’s Sesify

  • Ethereum wallet
  • Can run Ethereum apps in browser without running full Ethereum node

Salesforce’s Locker Service

  • One of primary co-authors of Realms and SES
  • Plugin platform

Caveats for Realms

  • Work in progress
  • Have to stringify to use
  • Still in draft

Q&A

  • What if add something bad? https://ocapjs.org/t/tofu-trusted-on-first-use-tool/27 Putting something bad in wrapper would show up in diff/code review.
  • How SES different than Object.freeze? Object.freeze only freezes that instance and doesn’t go up prototype chain 
  • How know what functions/authorities need to provide to packages? Still developing patterns of use. For now might be trial and error. Might need changes to module.
  • Why don’t we hear about npm install attacks in other languages? Still have problems. Java can’t protect [I raised my hand and described how Sonatype helps protect Maven Central]. Worse on JavaScript because lots of tiny packages. Visibility will help in future.
  • Will this be bolted on to web frameworks? Hasn’t yet, but hope will happen.

My impressions

While I was aware of the problem, the solution (or future solution) is really interesting! She left lots of time for Q&A which was nice after yesterday. [My track didn’t have much time for Q&A in most sessions]

[QCon 2019] Light the fire – how managers can spark new leaders

Posted on by
Reply

Nick Cadwell – Chief Product Officer at Looker (acquired by Google) previously Reedit, and Microsoft

For other QCon blog posts, see QCon live blog table of contents

Tech leads vs managers

  • Weren’t any managers
  • Lots of people called “tech leads” – nebulous job titles,
  • Split tech leads into managers and architects
  • Asked questions to determine if emphasis is people or tech. One question was about deadlines. That should be important to both.

Scaling

  • When growing, solving problems by hiring more people
  • Need more specialization and focus
  • Makes harder for people to switch
  • Creates politics/boundaries
  • Decreases ownership at individual level
  • People don’t feel empowered
  • Reduces productivity/retention/development/delegation. Recipe for burnout

Leadership vs Management

  • Management – stability, short team, plans around constraints
  • Leader- change, long term, sets/leads direction, long term
  • “Leadership is working with goals and vision. Management is working with objectives”
  • Position leadership doesn’t scale – don’t have to be a manager to lead.
  • Everyone is potentially a leader
  • People want opportunities to lead; not just positions.
  • Opportunity to lead outperforms financial incentives on retention surveys

Management tool: Leadership breadcrumbs

  • Provide visibility into problems that working on but don’t have capacity to fix
  • Invite discussions – solutions only, no complaining
  • Wait and repeat. Eventually someone will take the bait
  • Provide support and reward for stepping up
  • Don’t need to support every idea financial. Can just discuss further. The idea of taking it seriously matters

Individual tool: The Golden Question

  • Ask yourself or others what you would do if you had another pair of hands
  • These are opportunities for leadership

Leadership

  • Top leadership traits said to be: vision, empathy, empowerment, charisma and expertise
  • But there are counter examples
  • Steve Jobs wasn’t empowering
  • Jeff Bezos isn’t known for empathy
  • Elon Musk – burn out? should delegate more
  • Mark Zuckerberg not known for charisma
  • Passion is leadership fuel
  • Survey: Most millionaires started with goal of passion not money

Manager tool: Blue Flame Chart

  • Align individual passion and organizational needs
  • Intersect intrinsic/extrinsic motivators with org goals
  • Add team member feedback. Most individuals don’t see themselves clearly. Ask someone else what good at

Sparking leadership

  • First rule of leadership: everything is your fault
  • Biggest reason for not stepping up is fear of being blamed/help responsible
  • Switch mentality from contributor to leader
  • Leaders take responsiblity for what happens next

Mentor vs sponsor

  • Mentor – gives advice, makes suggestions, discusses
  • Sponsor – opens doors, shares hard feedback, pushes to strive for more, creates opportunities, advocates for you
  • [I think there is some overlap. Mentoring doesn’t just have to be about the positive. You can share hard feedback too..]

Code is a deprecicating asset, your network is an appreciating asset

My impressions

This was great. He has excellent passion and a clear relatable message. I suspect there are a lot of people in the room like me (who want to be leaders but not managers.) I was tired when I sat down and I’m not anymore. This keynote definitely woke up my brain!