Performance Profiling and Debugging

2022-06-17T00:00:00+00:00

Intro

Profiling and Debugging are complementary techniques used to discover facts about what a given system is doing and the performance characteristics of those actions’ . We will focus on .net debugging and profiling here, but in any given system typically uses databases, OS level services etc, and therefore you will probably need to debug or profile these systems as well. Profiling and debugging is huge areas, so this piece is focused on practical usage.

Debugging .net

Both .net (core) and .net Framework have extensive support for debugging. These tools can attach to a running process so you can step through the code as is runs. What detail you see are dependent on the build artifacts (debug vs release, symbols etc) You are not only limited to running processes. You can also open a memory dump, and these dumps can be taken by a range of tools. Finally Dotnet-trace can also connect to a running process, but as the name suggest only tracing the process.

Tools

Visual Studio
VS Code
Rider, dotTrace, dotMemory
ANTS Profiler /ANTS Memory Profiler
WinDBG /WinDbg Preview
Dotnet-dump / Dotnet-trace
Remote Debugger
…

Scenario: Exceptions being thrown by production system.

(If Permitted ) Get a memory dump. A memory dump will contain threads, stacks of treads, and heaps. All of these will both be managed and native, but our focus will be on the managed/.net side.

Alternatively, most IDEs support remote debugging. NB Secure this channel!

Start with Exception analysis. !analyze .foreach (ex {!dumpheap -type Exception -short}){.echo "********************************";!pe ${ex} }
Start with listing the current stacks. A good start hypothesis is that the active stacks in the dump will have recorded a hang. Check this out before moving on. ~* e !clrstack Lists stacks for all threads.
Your tool of choice may also automate checks for know issues like deadlocks. For WinDbg you can run !SyncBlk
Check runaway threads !runaway !threadpool

Profiling

Profiling is looking at how a system operates. This is typically used for performance reasons, as we assume that the behavior is as designed. Any code is generally either IO or CPU bound. Profiles may help you with both, but there are often differences in the tools and methods to profile between then. E.g. Jetbrains has dotTrace and dotMemory.

CPU Bound profiling

Here we typically look at algorithmic problems, measure and identify hotspots in the code. This can either be local or remote. Keep in mind that some profile methods may be very intrusive performance wise. This may not be acceptable in a production setting!

Memory Bound profiling

In managed/garbage collected runtimes like .net the memory is managed by the runtime, and as such you can only have a native memory leak. Such a leak can e.g. be a SqlConnection, File handle etc. Typically, a “leak” in .net is rather that something is holding on to a reference, and it is thus not collected. There is also another category that looks like a leak: not yet collected objects.

Very simplified: class object instances are allocated on the heap while value objects are on the stack. Also, while strings have some value like behavior, remember that they are stored on the heap, and any modification (substring etc) is in fact a copy, and may exist on the heap for an extended time period.

https://www.jetbrains.com/help/dotmemory/NET_Memory_Management_Concepts.html

Garbage Collector (GC) is the part of the .NET framework that allocates and releases memory for your .NET applications.
When a new process is started, the runtime reserves a region of address space for the process called the managed heap.
Objects are allocated in the heap contiguously one after another.
Memory allocation is a very fast process as it is just the adding of a value to a pointer.

Any object larger than 85KB are store separately on the Large Object Heap (LOH). This is not compacted during normal garbage collection. The LOH will thus get fragmented over time.

flowchart LR allocation(Allocation) subgraph gen0 direction TB 0_allocate_1(allocate some) --> 0_allocate_n(allocate more) --> 0_full(full) --> 0_GC_generation(Garbage collect Gen 0) end subgraph gen1 direction TB 1_allocate_1(moved into gen1) --> 1_allocate_n(moved even more into) --> 1_full(full) --> 1_GC_generation(Garbage collect Gen 1) end subgraph gen2 direction TB 2_allocate_1(moved into gen2) --> 2_allocate_n(moved even more into) --> 2_full(full) --> 2_GC_generation(Garbage collect Gen 2) end loh(Large Object Heap) allocation --> gen0 & loh gen0 -- still in use--> gen1 gen1 -- still in use--> gen2

Finding your roots

A key factor in memory pressure is holding on to references as short as possible. The sooner the last reference to an object is released the quicker an object can be garbage collected. Be cautious when adding delegates, event handlers etc to global or static scoped objects or methods. NB This also includes what singleton scoped dependency that is injected into your code. Most unwanted memory is due to this.

!DumpHeap -stat
!DumpHeap -stat -type SomeType.OrAnother

Rent, not allocate

For large allocation it will typically be much faster both on allocation as well as collection to rent an array from ArrayPool/ArrayPool.Shared. There are also more advanced allocation schemas that can be used like an Arena Allocator

Measure

Measure using Benchmarkdotnet, Rider Dynamic Program Analysis, DotMemory, Visual Studio etc. Go for large and frequent allocations and iterate.

My thoughts on Agile Organizations

2022-05-19T00:00:00+00:00

Intro

These reflections and observations are based on my experiences in multiple projects and organization and heavily inspired by Alex Cowan’s Digital Product Management course.

What is agile?

Agile is the realization that change is inevitable. An organization should optimize for change and make change easy as possible. This does not only apply to technical aspects. It is just as important- if not more- in the organizational structure. After all you cannot escape the organization:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. — Melvin E. Conway 1 2

Just having agile is of limited value if these interact and deliver in a non-agile organization. What are some indicators of such? · The area of responsibility vs control of change that a team has. · Forced inter-team communications/collaboration.

Responsibility vs control

The mismatch between responsibility and control often manifests with teams that are responsible for a thin slice of layers, whether one layer or a few layers of a value chain and does not control enough of the vertical slice. This leads to the team being dependent on other teams for its day-to-day business and significantly blocks changes.

Excessive forced inter-team communication

Forced in the meaning that the team needs resources outside the team to do its job/purpose. It will not be practical or even desirable to eliminate this completely. If this is excessive then it is an indicator that the organization may benefit from changes.

Solutions

What not to do

Don’t do a massive reorganization. Especially if this is against the will of the affected persons. Have we not learned that huge, big-bang releases are bad?

What to do

We should approach this scientifically in a hypothesis-driven process, where we acquire information: both facts and opinions from team members and stakeholders and together state hypotheses. These will need to be tested as isolated and on as small scale as possible.
For your next multi-team project or initiative: can you create a team (if only temporarily) or an integrated working group and try to verify the desirability, viability and feasibility of the proposed feature, project, or product.

Tomas Eilev Holthe-Tveit

Performance Profiling and Debugging

Intro

Debugging .net

Tools

Scenario: Exceptions being thrown by production system.

Profiling

CPU Bound profiling

Memory Bound profiling

Finding your roots

Rent, not allocate

Measure

My thoughts on Agile Organizations

Intro

What is agile?

Responsibility vs control

Excessive forced inter-team communication

Solutions

What not to do

What to do

Try, learn and improve!