Forem: Unicorn Developer

How to attract developers to your website with an IT events calendar

Unicorn Developer — Tue, 18 Apr 2023 15:01:11 +0000

How to advertise in an environment that doesn't welcome advertising.

Hi, this is the PVS-Studio team. We've been developing the code analyzer for C++, C#, and Java for 15 years, and now we have decided to launch an event calendar for IT professionals.

Context

Before we begin, a bit of context: our company sells the static code analyzer. This is a complex specific product that developers use to automatically search for errors and potential vulnerabilities. Business saves money because it is several times cheaper to fix an error at the stage of writing code than at the code testing stage (and even more so after the release).

PVS-Studio is used not only by thoroughbred IT companies: any company where one of the departments writes code subscribe to the license — gamedev, medical institutions, financial corporations.
One of our eternal goals is to increase the number of programmers who visit our website and know the usefulness of the analyzer.

Idea

Looking for new points of growth, we thought about how to attract more potential customers to the site, build up communication channels and distribute more product trials. A solution came from a problem that we ourselves encountered: among the bunch of websites, it is difficult to find a conference you want to speak at.

There is no common website: you search for C++ events in one place, and for Java in another. We had to keep a list of sources that need to be checked regularly, otherwise we risked not being able to apply for a major conference. And if you're just a participant, you can miss the ticket discounts end up in a situation where tickets are sold out.

We are sure that other developers face similar problems — that's why we have launched a common event calendar for developers. For users, this saves time and solves an urgent problem, for us it is an opportunity to highlight our main product.

Service

The first version of the service was released in March 2023. The website now has about four hundred conferences. You can filter conferences by country, subject, and format of the event. For example, the calendar can show which C++ conferences will be held in Germany this year in a couple of clicks.

Besides conferences for programmers, we add events to the website for IT managers, security specialists, game industry employees, and workers from other related fields. This is how we reach those who are not directly involved in coding, but may decide to buy PVS-Studio.

In addition, we try to cover small meetups: we believe that small communities are no less important than large international conferences.

Team

The main performer in the service is the event marketer. He spreads the news about the calendar on thematic sites, writes promo texts (like this one), orders advertising and arranges for advertising integrations. The web department draws unicorns and makes sure that nothing breaks on the website. One of the founders of the company follows the processes and vision.

The team that works on the calendar is united by one idea: to attract programmers to the website with the help of marketing, and then retain them through the benefits the service creates.

Promotion

It is easier and cheaper to advertise an aggregator than code analysis software. But the audience for these two products is very close.

The promotion of the service is in its early stages. In addition to this text, we will post articles on thematic websites for developers, make text posts in social media, buy advertising in telegram channels and search engines. We also actively communicate with IT chat members — they are eager to use interesting innovations. But you have to be careful in these conversations: it's easy to get banned.

We don't yet know how the programming community will welcome the service. Experienced developers often react extremely negatively to any advertising, so we will try to do delicate marketing. We hope that a free product with more or less understandable benefits will not be pelted with tomatoes.

Monetization

The service does not sell ads. We do not receive money for the publication of conferences. We add to the general list only those events that we consider important. As mentioned above, the purpose of the calendar is to put our brand in the infofield, not to make money from contextual advertising.

PVS-Studio has a B2B product and a fairly long sales cycle — several months. Therefore, here we primarily focus on the benefits that our calendar will bring to the user. Maybe one day this user will want to buy our analyzer, ask for a trial or tell their colleague about the program, the one who just codes in C++.

We will see which part of the users who use the calendar will try out our code analyzer.

Welcome: event calendar for developers.

Find your conference: calendar for developers

Unicorn Developer — Fri, 07 Apr 2023 07:44:00 +0000

Finding a conference to your liking is a challenge. Many people are probably familiar with the situation: rummaging through a bunch of websites, stacking on the search engines and thematic communities. You have to bookmark a bunch of sources where announcements of events are published randomly. One website only posts JS conferences, while another only posts about Android development meetups. It's inconvenient. Therefore, the PVS-Studio team launched an event aggregator for developers. You can find it on our website.

Now all IT events are collected in one place. You no longer have to tediously search for a conference where you want to make a presentation or just be a participant.

Welcome: Events for developers!

The events are sorted by subject, country, and the format. The aggregator will show you a catalog of all C++ conferences or Java meetings in Germany in a couple of clicks. You don't have to scroll through a long overall list.

The card of each event contains information about the date, the venue, the format, and the event organizer. There are also tags that indicate the topic of the conference. All conferences are divided by the programming language and industry. Events on information security, mobile development, gamedev, and other topics have their own categories.

The aggregator has now collected around 300 events. Each conference is checked before getting into the list. Only the most useful events reach the user. Once a week, the team searches for the newest events and adds them to the catalog. You can suggest your conference through the feedback form.

In addition to large conferences known around the world, the aggregator can show small meetups. This is one of the key features of our service — we want to promote small communities of developers from different countries who might find it difficult to find an audience without third-party support. When you attend them, it's easy to meet really passionate professionals: these guys are boosting the IT industry with their enthusiasm.

The service operates without advertising: PVS-Studio doesn't get paid for publishing conferences and only includes events that are considered important.

The aggregator contains events for many countries, but the focus is on English-speaking conferences. We believe that developers from all over the world will use this service.

The service is brand new — the team shared it to the public in March 2023. It would be cool if you could share your feedback :)

Top 10 bugs found in C++ projects in 2022

Unicorn Developer — Fri, 27 Jan 2023 11:59:24 +0000

New Year is coming! It means, according to tradition, it's time to recall 10 of the most interesting warnings that PVS-Studio found during 2022.

It is worth noting that this year there were not as many articles about project checks as in past years. Articles on our blog have become more varied, and translations of articles by guest authors have also appeared. However, I, as one of our blog's readers (and as the author of some articles from it), still made the top list of bugs found by the PVS-Studio analyzer that are described in our articles on project checks. Let me point out that this top is rather subjective — it has many bugs from articles written by me :).

If you, dear readers, have your own vision of how this top should look, please, share it in the comments. So, let's get started, enjoy reading!

Tenth place: A classic typo

V501 [CWE-570] There are identical sub-expressions '(SrcTy.isPointer() && DstTy.isScalar())' to the left and to the right of the '||' operator. CallLowering.cpp 1198

static bool isCopyCompatibleType(LLT SrcTy, LLT DstTy) 
{
  if (SrcTy == DstTy)
    return true;

  if (SrcTy.getSizeInBits() != DstTy.getSizeInBits())
    return false;

  SrcTy = SrcTy.getScalarType();
  DstTy = DstTy.getScalarType();

  return (SrcTy.isPointer() && DstTy.isScalar()) ||
         (DstTy.isScalar() && SrcTy.isPointer());
}

We see a classic typo in this code fragment:

(SrcTy.isPointer() && DstTy.isScalar()) ||
(DstTy.isScalar()  && SrcTy.isPointer())

Here is the error:

(SrcTy.isPointer() && DstTy.isScalar()) ||
(DstTy.isScalar()  && SrcTy.isPointer())

A developer simultaneously changed Src to Dst and Pointer to Scalar in the second line. As a result, it turns out that the same thing is checked twice! The code above is equivalent to:

(SrcTy.isPointer() && DstTy.isScalar()) ||
(SrcTy.isPointer() && DstTy.isScalar())

The correct option:

(SrcTy.isPointer() && DstTy.isScalar()) ||
(SrcTy.isScalar()  && DstTy.isPointer())

This error entered the top of the article: "Examples of errors that PVS-Studio found in LLVM 15.0."

Ninth place: Where would we be without array overrun

V557 Array overrun is possible. The 'j' index is pointing beyond array bound. OgreAnimationTrack.cpp 219

void AnimationTrack::_buildKeyFrameIndexMap(
  const std::vector<Real>& keyFrameTimes)
{

  // ....

  size_t i = 0, j = 0;
  while (j <= keyFrameTimes.size())                    // <=
  {
    mKeyFrameIndexMap[j] = static_cast<ushort>(i);
    while (i < mKeyFrames.size()
      && mKeyFrames[i]->getTime() <= keyFrameTimes[j]) // <=
      ++i;
    ++j;
  }
}

The j index that gives us access to the elements of the keyFrameTimes container is incremented to a value equal to the container size. We can fix it:

while (j < keyFrameTimes.size())
{
  // ....
}

I took this error from the article: "Checking the Ogre3D framework with the PVS-Studio static analyzer."

Eighth place: A clear example of a null pointer dereference

V522 [CERT-EXP34-C] Dereferencing of the null pointer 'document' might take place. TextBlockCursor.cpp 332

auto BlockCursor::begin() -> std::list<TextBlock>::iterator
{
  return document == nullptr 
            ? document->blocks.end() : document->blocks.begin();
}

I remember that the first time I saw this piece of code, I couldn't help but facepalm. Let's figure out what's going on here. We see an explicit check of the document pointer for nullptr and its dereference in both branches of the ternary operator.

The code is correct only if the developer aimed to crash the program.

I took this error from the article: "MuditaOS: Will your alarm clock go off? Part I".

Seventh place: Learning to count to seven

V557 [CWE-787] Array overrun is possible. The 'dynamicStateCount ++' index is pointing beyond array bound. VltGraphics.cpp 157

VkPipeline VltGraphicsPipeline::createPipeline(/* .... */) const
{
  // ....
  std::array<VkDynamicState, 6> dynamicStates;
  uint32_t                      dynamicStateCount = 0;
  dynamicStates[dynamicStateCount++] = VK_DYNAMIC_STATE_VIEWPORT;
  dynamicStates[dynamicStateCount++] = VK_DYNAMIC_STATE_SCISSOR;
  if (state.useDynamicDepthBias())
    dynamicStates[dynamicStateCount++] = VK_DYNAMIC_STATE_DEPTH_BIAS;
  if (state.useDynamicDepthBounds())
  {
    dynamicStates[dynamicStateCount++] = VK_DYNAMIC_STATE_DEPTH_BOUNDS;
    dynamicStates[dynamicStateCount++] =
                             VK_DYNAMIC_STATE_DEPTH_BOUNDS_TEST_ENABLE;
  }
  if (state.useDynamicBlendConstants())
    dynamicStates[dynamicStateCount++] = VK_DYNAMIC_STATE_BLEND_CONSTANTS;
  if (state.useDynamicStencilRef())
    dynamicStates[dynamicStateCount++] = VK_DYNAMIC_STATE_STENCIL_REFERENCE;
  // ....
}

The analyzer warns that an overflow of the dynamicStates array may occur. There are 4 checks in this code fragment:

if (state.useDynamicDepthBias())
if (state.useDynamicDepthBounds())
if (state.useDynamicBlendConstants())
if (state.useDynamicStencilRef())

Each of these checks is a check of one of the independent flags. For example, the check of if (state.useDynamicDepthBias()):

bool useDynamicDepthBias() const
{
  return rs.depthBiasEnable();
}

VkBool32 depthBiasEnable() const
{
  return VkBool32(m_depthBiasEnable);
}

It turns out that all these 4 checks can be true at the same time. Then 7 lines of the* 'dynamicStates[dynamicStateCount++] =....'* kind will be executed. On the seventh such line, there will be a call to dynamicStates[6]. It's an array index out of bounds.

This bug is from the top of the article: "Checking the GPCS4 emulator: will we ever be able to play "Bloodborne" on PC?."

Sixth place: Throwing an exception from the noexcept function

V509 [CERT-DCL57-CPP] The noexcept function '=' calls function 'setName' which can potentially throw an exception. Consider wrapping it in a try..catch block. Device.cpp 48

struct Device
{
  static constexpr auto NameBufferSize = 240;
  ....
  void setName(const std::string &name)
  {
    if (name.size() > NameBufferSize) 
    {
        throw std::runtime_error("Requested name is bigger than buffer 
                                  size");
    }
    strcpy(this->name.data(), name.c_str());
  }
  ....
}

....

Devicei &Devicei::operator=(Devicei &&d) noexcept
{
  setName(d.name.data());
}

Here the analyzer detected that a function, marked as noexcept, calls a function that throws an exception. If an exception arises from the nothrow function's body, the nothrow function calls std::terminate, and the program crashes.

It could make sense to wrap the setName function in the function-try-block and process the exceptional situation there — or one could use something else instead of generating the exception.

I took this error from the article: "MuditaOS: Will your alarm clock go off? Part I".

Fifth place: Incorrect work with dynamic memory

PVS-Studio issued two warnings at once for the code snippet presented below:

V611 [CERT-MEM51-CPP] The memory was allocated using 'new T[]' operator but was released using the 'delete' operator. Consider inspecting this code. It's probably better to use 'delete [] heightfieldData;'. PhysicsServerCommandProcessor.cpp 4741
V773 [CERT-MEM31-C, CERT-MEM51-CPP] The function was exited without releasing the 'worldImporter' pointer. A memory leak is possible. PhysicsServerCommandProcessor.cpp 4742

bool PhysicsServerCommandProcessor::processCreateCollisionShapeCommand(....)
{
  btMultiBodyWorldImporter* worldImporter = new btMultiBodyWorldImporter(....);
  ....
  const unsigned char* heightfieldData = 0;
  ....
  heightfieldData = new unsigned char[width * height * sizeof(btScalar)];
  ....
  delete heightfieldData;
  return ....;
}

Let's start with the V773 warning. The memory for the worldImporter *pointer was allocated using the *new operator and was not released upon exiting the function.

Let's move on to the V611warning and the heightfieldData buffer. Here, the developer cleared the dynamically allocated memory, but did so in the wrong way. Instead of calling delete[] to release previously allocated memory with the new[] operator, they called a simple delete. According to the standard, such code will lead to undefined behavior — here is a link to the corresponding item.

By the way, you can read about why arrays need to be deleted by delete[], and about how it all works in general in article written by my colleague.

Here is the fixed version:

bool PhysicsServerCommandProcessor::processCreateCollisionShapeCommand(....)
{
  btMultiBodyWorldImporter* worldImporter = new btMultiBodyWorldImporter(....);
  ....
  const unsigned char* heightfieldData = 0;
  ....
  heightfieldData = new unsigned char[width * height * sizeof(btScalar)];
  ....

  delete   worldImporter;
  delete[] heightfieldData;
  return ....;
}

Also, it's possible to avoid problems with manual memory cleanup by writing more modern code. For example, automatically free up memory by using std::unique_ptr. The code is shorter and more reliable. It will protect against unreleased memory errors in case of early exit from the function:

bool PhysicsServerCommandProcessor::processCreateCollisionShapeCommand(....)
{
  auto worldImporter = std::make_unique<btMultiBodyWorldImporter> ();
  ....
  std::unique_ptr<unsigned char[]> heightfieldData;
  ....
  heightfieldData = std::make_unique_for_overwrite<unsigned char[]>
                                (width * height * sizeof(btScalar));
  ....
  return ....;
}

You can find this error in the article: "In the world of anthropomorphic animals: PVS-Studio checked Overgrowth".

Fourth place: The power of Symbolic execution

V560 [CWE-570] A part of conditional expression is always false: DefaultCC == ToCC. SemaType.cpp 7856

void Sema::adjustMemberFunctionCC(QualType &T, bool IsStatic, bool IsCtorOrDtor,
                                  SourceLocation Loc) {
  ....
  CallingConv CurCC = FT->getCallConv();
  CallingConv ToCC = Context.getDefaultCallingConvention(IsVariadic, !IsStatic);

  if (CurCC == ToCC)
    return;
  ....
  CallingConv DefaultCC = 
    Context.getDefaultCallingConvention(IsVariadic, IsStatic);

  if (CurCC != DefaultCC || DefaultCC == ToCC)
    return;
  ....
}

Let's figure out why the analyzer decided that the right part of the condition is always false. For convenience, let's remove all superfluous details and replace the names:

A = ....;
B = ....;
if (A == B) return;
C = ....;
if (A != C || C == B)

Let's look at how this code works:

we have 3 variables A, B, C and values are unknown to us;
but we know that if A == B, there is an exit from the function;
therefore, if the function continues to execute, then A != B;
if A != C, then, due to the short-circuit evaluation, the right subexpression is not evaluated;
if the right subexpression "C == B" is evaluated, then A == C;
if A != B and A == C, then C cannot be equal to B.

This error entered the top of the article: "Examples of errors that PVS-Studio found in LLVM 15.0."

Third place: hmm, how we love std::optional

A couple of analyzer warnings:

V571 Recurring check. The 'if (activeInput)' condition was already verified in line 249. ServiceAudio.cpp 250
V547 Expression 'activeInput' is always true. ServiceAudio.cpp 250

std::optional<AudioMux::Input *> AudioMux::GetActiveInput();

....

auto Audio::handleSetVolume(....) -> std::unique_ptr<AudioResponseMessage>
{
  ....
  if (const auto activeInput = audioMux.GetActiveInput(); activeInput) 
  {
    if (activeInput) 
    {
      retCode = activeInput.value()->audio->SetOutputVolume(clampedValue);
    }
  }
  ....
}

activeInput is an std::optional entity from the pointer to AudioMux::Input. The nested if statement contains the value *member function call.* The function is guaranteed to return the pointer and will not throw an exception. After, the result is dereferenced.

However, the function may return either a valid — or a null pointer. The plan for the nested if statement was probably to check this pointer. Hm, I also like wrapping pointers and boolean values in std::optional! And then going through the same grief each time :).

Let's see how we can fix it:

std::optional<AudioMux::Input *> AudioMux::GetActiveInput();

....

auto Audio::handleSetVolume(....) -> std::unique_ptr<AudioResponseMessage>
{
  ....
  if (const auto activeInput = audioMux.GetActiveInput(); activeInput) 
  {
    if (*activeInput) 
    {
      retCode = (*activeInput)->audio->SetOutputVolume(clampedValue);
    }
  }
  ....
}

I took this error from the article: "MuditaOS: Will your alarm clock go off? Part I".

Second place: Incorrect use of the flag

V547 [CWE-570] Expression 'nOldFlag & VMPF_NOACCESS' is always false. PlatMemory.cpp 22

#define PAGE_NOACCESS           0x01
#define PAGE_READONLY           0x02
#define PAGE_READWRITE          0x04
#define PAGE_EXECUTE            0x10
#define PAGE_EXECUTE_READ       0x20
#define PAGE_EXECUTE_READWRITE  0x40

enum VM_PROTECT_FLAG
{
  VMPF_NOACCESS  = 0x00000000,
  VMPF_CPU_READ  = 0x00000001,
  VMPF_CPU_WRITE = 0x00000002,
  VMPF_CPU_EXEC  = 0x00000004,
  VMPF_CPU_RW    = VMPF_CPU_READ | VMPF_CPU_WRITE,
  VMPF_CPU_RWX   = VMPF_CPU_READ | VMPF_CPU_WRITE | VMPF_CPU_EXEC,
};

inline uint32_t GetProtectFlag(VM_PROTECT_FLAG nOldFlag)
{
  uint32_t nNewFlag = 0;
  do
  {
    if (nOldFlag & VMPF_NOACCESS)
    {
      nNewFlag = PAGE_NOACCESS;
      break;
    }

    if (nOldFlag & VMPF_CPU_READ)
    {
      nNewFlag = PAGE_READONLY;
    }

    if (nOldFlag & VMPF_CPU_WRITE)
    {
      nNewFlag = PAGE_READWRITE;
    }

    if (nOldFlag & VMPF_CPU_EXEC)
    {
      nNewFlag = PAGE_EXECUTE_READWRITE;
    }

  } while (false);
  return nNewFlag;
}

The GetProtectFlag function converts a flag with file access permission from one format to another. However, the function does this incorrectly. The developer did not take into account that the value of VMPF_NOACCESS is zero. Because of this, the if (nOldFlag & VMPF_NOACCESS) condition is always false and the function will never return the PAGE_NOACCESS value.

In addition, the GetProtectFlag function incorrectly converts not only the VMPF_NOACCESS flag, but also other flags. For example, the VMPF_CPU_EXEC flag will be converted to the PAGE_EXECUTE_READWRITE flag.

When a developer who wrote the article was thinking how to fix that issue, his first thought was to write something like this:

inline uint32_t GetProtectFlag(VM_PROTECT_FLAG nOldFlag)
{
  uint32_t nNewFlag = PAGE_NOACCESS;
  if (nOldFlag & VMPF_CPU_READ)
  {
    nNewFlag |= PAGE_READ;
  }

  if (nOldFlag & VMPF_CPU_WRITE)
  {
    nNewFlag |= PAGE_WRITE;
  }

  if (nOldFlag & VMPF_CPU_EXEC)
  {
    nNewFlag |= PAGE_EXECUTE;
  }

  return nNewFlag;
}

However, in this case, this approach does not work. The thing is, PAGE_NOACCESS, PAGE_READONLY, and other flags are Windows flags and they have their own specifics. For example, there is no PAGE_WRITE *flag among them. It is assumed that if there are write permissions, then at least there are also read permissions. For the same reasons, there is no *PAGE_EXECUTE_WRITE flag.

In addition, the bitwise "OR" of two Windows flags does not result in a flag corresponding to the sum of the permissions: PAGE_READONLY | PAGE_EXECUTE != PAGE_EXECUTE_READ. Therefore, you need to iterate through all possible flag combinations:

inline uint32_t GetProtectFlag(VM_PROTECT_FLAG nOldFlag)
{
  switch (nOldFlag)
  {
    case VMPF_NOACCESS:
      return PAGE_NOACCESS;
    case VMPF_CPU_READ:
      return PAGE_READONLY;
    case VMPF_CPU_WRITE: // same as ReadWrite
    case VMPF_CPU_RW:
      return PAGE_READWRITE;
    case VMPF_CPU_EXEC:
      return PAGE_EXECUTE;
    case VMPF_CPU_READ | VMPF_CPU_EXEC:
      return PAGE_EXECUTE_READ:
    case VMPF_CPU_WRITE | VMPF_CPU_EXEC: // same as ExecuteReadWrite
    case VMPF_CPU_RWX:
      return PAGE_EXECUTE_READWRITE;
    default:
      LOG("unknown PS4 flag");
      return PAGE_NOACCESS;
  }
}

This bug is from the article: "Checking the GPCS4 emulator: will we ever be able to play "Bloodborne" on PC?."

First place: PVS-Studio prevents rash code changes!

V530 [CWE-252]: The return value of function 'clamp' is required to be utilized. BLI_math_vector.hh 88

So, once upon a time there was code that processed a vector of values. It prevented values from going beyond a certain range.

#define CLAMP(a, b, c) \
  { \
    if ((a) < (b)) { \
      (a) = (b); \
    } \
    else if ((a) > (c)) { \
      (a) = (c); \
    } \
  } \
  (void)0

template <typename T> inline T
clamp(const T &a, const bT &min_v, const bT &max_v)
{
  T result = a;
  for (int i = 0; i < T::type_length; i++) {
    CLAMP(result[i], min_v, max_v);
  }
  return result;
}

Everything was good. And then a developer decided to abandon the custom CLAMP macro and use the standard std::clamp function. And the commit that supposed to make the code better looked like this:

template <typename T, int Size>
inline vec_base<T, Size>
  clamp(const vec_base<T, Size> &a, const T &min, const T &max)
{
  vec_base<T, Size> result = a;
  for (int i = 0; i < Size; i++) {
    std::clamp(result[i], min, max);
  }
  return result;
}

It seems like the developer was in a hurry. Do you see the error? Maybe yes, maybe no. Anyway, the developer who wrote the code, didn't notice that it was broken.

The point being — the std::clamp function doesn't change the value of the element in the container:

template <class T>
constexpr const T&
clamp( const T& v, const T& lo, const T& hi );

The CLAMP macro used to change the value, but the standard function did not. Now the code is broken and is waiting for someone to notice an error and look for its cause.

Let me point out that we regularly check several open-source projects. Sometimes it allows you to write quick articles and show interesting bugs in code just written.

Although we already written many articles on this topic, I described the warning that seemed the most striking for me. By the way, if you haven't read these articles, then I strongly recommend you do!

As the author of this compilation, I want to give the first place not only to this error, but to the whole series of articles on the topic. In fact, this is a great work of one well-known person (Andrey, hi!) who started a separate genre of articles on our blog that delighted us throughout the year 2022.

The same error entered the top of the article: "How PVS-Studio prevents rash code changes, example N4".

Conclusion

This year was not as prolific in terms of articles on C++ project checks as the past ones. However, I hope that we were still able to delight you with a selection of interesting bugs. We have also written many articles in other genres this year. You can find them on our blog.

It has become a tradition to post such articles on New Year's Eve, so here are the articles with the top 10 bugs found in C and C++ projects of the past years: 2016, 2017, 2018, 2019, 2020, 2021.

Help the compiler, and the compiler will help you. Subtleties of working with nullable reference types in C#

Unicorn Developer — Tue, 24 Jan 2023 07:35:37 +0000

Nullable reference types appeared in C# 3 years ago. By this time, they found their audience. But even those who work with this "beast" may not know all its capabilities. Let's figure out how to work with these types more efficiently.

Introduction

Nullable reference types are designed to help create a better and safer application architecture. At the code writing stage, it is necessary to understand whether this or that reference variable can be null or not, whether the method can return null, and so on.

It is safe to say that every developer has encountered NRE (NullReferenceException). And the fact that this exception can be generated at the development stage is a good scenario, because you can fix the problem immediately. It is much worse when the user finds the problem when working with the product. Nullable reference types help protect against NRE.

In this article I will talk about a number of non-obvious features related to nullable reference types. But it's worth starting with a brief description of these types.

A word about nullable reference

In terms of program execution logic, a nullable reference type is no different from a reference type. The difference between them is only in the specific annotation that the first one has. The annotation allows the compiler to conclude whether a particular variable or expression can be null. To use nullable reference types, you need to make sure that the nullable context is enabled for the project or file (I will describe later how to do this).

To declare a nullable reference variable, add '?' at the end of the type name.

Example:

string? str = null;

Now the variable str can be null, and the compiler will not issue a warning for this code. If you don't add '?' when declaring a variable and assigning it with null, a warning will be issued.

It is possible to suppress compiler warnings about possible writing null to a reference variable that is not marked as nullable.

Example:

object? GetPotentialNull(bool flag)
{
  return flag ? null : new object();
}

void Foo()
{
  object obj = GetPotentialNull(false);
}

The obj variable will never be assigned with null, but the compiler does not always understand this. You can suppress the warning as follows:

object obj = GetPotentialNull(false)!;

Using the '!' operator, we "tell" the compiler that the method will definitely not return null. Therefore, there will be no warnings for this code fragment.

The functionality available when working with nullable reference types is not limited to declaring variables of that type (using '?') and suppressing warnings with '!'. Below I'll look at the most interesting features when working with nullable reference types.

Working with a nullable context

There are a number of mechanisms for more flexible work with nullable reference types. Let's look at some of them.

Working with attributes

Attributes can be used to tell the compiler the null-state of various elements. Let's look at the most interesting ones. Check out the documentation to find the full list of attributes.

To make it easier, let's introduce the term — null-state. The null-state is information about whether a variable or expression can be null at a given time.

AllowNull

Let's look how the attribute work. Here is an example:

public string Name
{
  get => _name;
  set => _name = value ?? "defaultName";
}

private string _name;

If you write the null value to the Name property, the compiler will issue a warning: Cannot convert null literal to non-nullable reference type. But you can see from the implementation of the property that it can be null. In this case, the "defaultName" string is assigned to the _name field.

If you add '?' to the property type, the compiler will assume that:

the set accessor can accept null (this is correct);
the get accessor can return null (this is an error).

For correct implementation, it is worth adding the AllowNull attribute to the property:

[AllowNull]
public string Name

After that, the compiler will assume that Name may be assigned with null, although the property's type is not marked as nullable. If you assign the value of this property to a variable that should never be null, then there will be no warnings.

NotNullWhen

Suppose we have a method that checks a variable for null. Depending on the result of this check, the method returns a value of the bool type. This method informs us about the null-state of the variable.

Here's a synthetic code example:

bool CheckNotNull(object? obj)
{
  return obj != null;
}

This method checks the obj parameter for null and returns a value of the bool type depending on the check result.

Let's use the result of this method in the condition:

public void Foo(object? obj1)
{
  object obj2 = new object();

  if (CheckNotNull(obj1))
    obj2 = obj1;
}

The compiler will issue a warning to code above: Converting null literal or possibly null value to non-nullable type. But such a scenario is impossible, since the condition guarantees that obj1 is not null in the then branch. The problem is that the compiler doesn't understand this, so we have to help it.

Let's change the signature of the CheckNotNull method by adding the NotNullWhen attribute:

bool CheckNotNull([NotNullWhen(true)]object? obj)

This attribute takes a value of the bool type as the first argument. With NotNullWhen, we link the null-state of the argument with the return value of the method. In this case, we "tell" the compiler that if the method returns true, the argument has a value other than null.

There is a peculiarity associated with this attribute.

Here are some examples:

Using the out modifier

bool GetValidOrDefaultName([NotNullWhen(true)] out string? validOrDefaultName, 
                           string name)
{
  if (name == null)
  {
    validOrDefaultName = name;
    return true;
  }
  else
  {
    validOrDefaultName = "defaultName";
    return false;
  }
}

Here, the compiler will issue a warning: Parameter 'validOrDefaultName' must have a non-null value when exiting with 'true'. It is quite reasonable, since '==' is used in the condition instead of the '!=' operator. In this implementation, the method returns true when validOrDefaultName is null.

Using the ref modifier

bool SetDefaultIfNotValid([NotNullWhen(true)] ref string? name)
{
  if (name == null)
    return true;

  name = "defaultName";
  return false;
}

We will also get a warning for this code fragment: Parameter 'name' must have a non-null value when exiting with 'true'. Similarly to the previous example, the warning is reasonable. '==' is used instead of the '!=' operator.

Without using a modifier

bool CheckingForNull([NotNullWhen(true)] string? name)
{
  if (name == null)
    return true;

  Console.WriteLine("name is null");
  return false;
}

The situation here is similar to previous cases. If name equals null, the method returns true. Following the logic of previous examples, a warning should also be issued here: Parameter 'name' must have a non-null value when exiting with 'true'. However, there is no warning. It's hard to say what's caused this, but it looks strange.

NotNullIfNotNull

This attribute allows you to establish a relationship between the argument and the return value of the method. If the argument is not null, the return value is also not null, and vice versa.

Example:

public string? GetString(object? obj)
{
  return obj == null ? null : string.Empty;
}

The GetString method returns null or an empty string, depending on the null-state of the argument.

Usage of this method:

public void Foo(object? obj)
{
  string str = string.Empty;

  if(obj != null)
    str = GetString(obj);
}

Compiler's warning for this code: Converting null literal or possibly null value to non-nullable type. In this case, the compiler is lying. Assignment is performed in the body of if, the condition of which guarantees that GetString will not return null. To help the compiler, let's add the NotNullIfNotNull attribute for the return value of the method:

[return: NotNullIfNotNull("obj")]
public string? GetString(object? obj)

Note. Starting with C#11, you can get the parameter name using the nameof expression*.* In this case, it would be nameof(obj).

The NotNullIfNotNull attribute takes the value of the string type as the first argument — the name of the parameter, based on which the null-state of the return value is set. Now the compiler has information about the relationship between obj and the return value of the method: if obj is not null, the return value of the method will not be null, and vice versa.

MemberNotNull

Let's start with an example:

class Person
{
  private string _name;

  public Person()
  {
    SetDefaultName();
  }

  private void SetDefaultName()
  {
    _name = "Bob";
  }
}

The compiler will issue a warning to this code fragment:* Non-nullable field '_name' must contain a non-null value when exiting constructor. Consider declaring the field as nullable*. However, the SetDefaultName method is called in the constructor's body, which initializes the only field of the class. This means that the compiler's message is false. The MemberNotNull attribute allows you to solve the problem:

[MemberNotNull(nameof(_name))]
private void SetDefaultName()

This attribute takes an argument of the string[] type with the params keyword. The strings need to match the names of the members that are initialized in the method.

Thus, we are indicating that the value of the* _name* field will not be null after this method is called. Now the compiler can understand that the field is initialized in the constructor.

MemberNotNullWhen

Let's look at the example:

class Person
{
  static readonly Regex _nameReg = new Regex(@"^I'm \w*");

  private string _name;

  public Person(string name)
  {
    if (!TryInitialize(name))
      _name = "invalid name";
  }

  private bool TryInitialize(string name)
  {
    if (_nameReg.IsMatch(name))
    {
      _name = name;
      return true;
    }
    else
      return false;
  }
}

TryInitialize will initialize _name if the argument's value matches some pattern. The method returns true when the field has been initialized, otherwise it returns false. Depending on the result of executing TryInitialize, a value is assigned to the _name field in the constructor. In this implementation, _name cannot be not initialized in the constructor. However, the compiler will issue a warning: Non-nullable field '_name' must contain a non-null value when exiting constructor. Consider declaring the field as nullable.

To fix the situation, you need to add the MemberNotNullWhen attribute:

[MemberNotNullWhen(true, nameof(_name))]
private bool TryInitialize(string name)

The type of the first argument is* bool*, the second argument's type is string[] (with the params keyword). The attribute is used for methods with a return value of the* bool type. The logic is simple: if the method returns a value that corresponds to the first argument of the attribute, the class members passed to *params will be considered initialized.

DoesNotReturn and DoesNotReturnIf

It is not uncommon to have to create methods that throw out exceptions if something has not gone according to plan. Unfortunately, the compiler cannot always understand that program execution will be terminated after such a method is called.

Example:

private void ThrowException()
{
  throw new Exception();
}

void Foo(string? str)
{
  if (str == null)
    ThrowException();

  string notNullStr = str;
}

For code above, the compiler will issue a warning: Converting null literal or possibly null value to non-nullable type. However, if str is null, the execution of the method will not reach the code fragment with the assignment, as an exception will be thrown. Thus, at the time of assignment, the str variable cannot be null.

The DoesNotReturn attribute allows you to tell the compiler that after executing the method marked with the attribute, the execution of the calling method stops.

Let's add the attribute for the throwException:

[DoesNotReturn]
private void ThrowException()

Now the compiler knows that after this method is called, control will not be returned to the calling method. Therefore, null will never be written to notNullStr.

The DoesNotReturnIf attribute works similarly to DoesNotReturn, except for checking an additional condition.

Example:

private void ThrowException([DoesNotReturnIf(true)] bool flag)
{
  if(flag)
    throw new Exception();
}

The compiler will assume that throwException will not return control to the calling method if the flag parameter is set to true.

Specifying context at the project level

To change the nullable context at the project level, you need to open the project properties and select the context in the "Build" section.

You can set the nullable context in the project file (.csproj). You need to open this file and write the value to the Nullable property:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>disable</Nullable>             //<=
  </PropertyGroup>
</Project>

It is likely that many people know that you can turn on or turn off the nullable context. However, there are two more context options.

Warnings

Behavior in the nullable warning context:

the '?' sign does not affect the analysis in any way;
from the point of view of the compiler, all values of the reference type can be null by default;
if you write the '?' sign, the compiler will issue a warning that it should not be used in this context;
the compiler will issue a warning only for those parts of code where the null reference is dereferenced;
you can indicate that the expression is not null using the '!' operator.

This mode helps protect against exceptions of NullReferenceException type. The mode informs about the dereference of null reference.

Annotations

Behavior in the nullable annotation context:

there are no warnings related to dereference of null references and errors when working with nullable reference;
the compiler does not issue warnings when '?' and '!' are used.

This mode helps make a smooth entry into the use of nullable reference types in the project. It allows you to markup variables that can and cannot be null.

Working with preprocessor directives

Preprocessor directives are used at the file level with the .cs extension and allow you to change the states of the nullable context for fragments of code in this file. The way it works is similar to that described in the previous section. Each directive starts with '#'.

Let's look at all possible directives:

#nullable disable – disables the nullable context;
#nullable enable – enables nullable context;
#nullable restore – restores the nullable context to its value at the project level;
#nullable disable annotations – disables annotation context;
#nullable enable annotations – enables annotation context;
#nullable restore – restores the nullable context to its value at the project level;
#nullable disable warnings – disables the warning context;
#nullable enable warnings – enables the warning context;
#nullable restore warnings – restores the warning context to its value at the project level.

In fact, the enable value represents the enabled context of annotations and the context of warnings, and disable – on the contrary, these same contexts are in the disabled state. So the '#nullable enable' directive would be equivalent to writing '#nullable enable annotations' and '#nullable enable warnings' together.

You can use multiple directives in one file at once. This allows you to set a different nullable context for different code fragments.

Let's look at an example of such usage (at the project level, nullable-context is disabled):

.... // nullable-context is disabled in this code fragment 
#nullable enable warnings
.... // the warning context is enabled in this code fragment
#nullable enable annotations
.... // the context of warnings and annotations is enabled 
     // in this code fragment
#nullable disable annotations
.... // only the warning context is enabled in this code fragment
#nullable restore
.... // nullable-context is disabled in this code fragment  
     // (since the Nullable property – disable)

Conclusion

In conclusion, being able to use nullable reference types should be of great benefit to developers. These types allow you to make the application more secure and correct from the point of view of architecture.

This mechanism is not without its drawbacks either. About drawbacks, and in general about nullable reference types, my colleagues told in articles: one, two. The ability to add attributes makes sense largely because of the imperfection of the static analyzer. Therefore, it is necessary to add annotations to methods, fields, etc. manually, because the analyzer cannot understand some relationships. For example, the relationship between the return value of a method and the null state of a variable.

A number of drawbacks are the result of insufficient in-depth analysis. Such analysis cannot be done on the fly. On the other hand, it is not required. nullable-context is a good help in the code-writing process. When part of the functionality is ready and it needs to be tested, we recommend using tools for deeper analysis – for example, PVS-Studio.

Top 10 bugs found in C# projects in 2022

Unicorn Developer — Mon, 16 Jan 2023 11:55:03 +0000

In 2022, the PVS-Studio developers wrote lots of articles where they described bugs found in open-source projects. Now it's time to choose the most interesting ones.

How was this top created?

I looked through all C# articles published this year and inspected all the bugs described there. To make this top more diverse, I used the following criteria:

one project — one error;
errors should be different from each other.

Now let's see what we've got.

10th place. Attempt to unsubscribe in Stride

A classic analyzer warning takes the tenth place today. This error was found during the Stride Game Engine project check:

private void RemoveEditor(....)
{
  ....
  if (....)
  {
    multiEditor.OpenedAssets.CollectionChanged -= (_, e) => 
      MultiEditorOpenAssetsChanged(multiEditor, e);
    ....
  }
  ....
}

V3084. Anonymous function is used to unsubscribe from 'CollectionChanged' event. No handlers will be unsubscribed, as a separate delegate instance is created for each anonymous function declaration. AssetEditorsManager.cs 444

Judging by the analyzer warning, the code doesn't unsubscribe from the event. Will it cause trouble? It can!

9th place. Ambiguous assignment in Bitwarden

The ninth place goes to an error from the Bitwarden project check. Let's look at it:

public class BillingInvoice
{
  public BillingInvoice(Invoice inv)
  {
    Amount = inv.AmountDue / 100M;      // <=
    Date = inv.Created;
    Url = inv.HostedInvoiceUrl;
    PdfUrl = inv.InvoicePdf;
    Number = inv.Number;
    Paid = inv.Paid;
    Amount = inv.Total / 100M;          // <=
  }
  public decimal Amount { get; set; }
  public DateTime? Date { get; set; }
  public string Url { get; set; }
  public string PdfUrl { get; set; }
  public string Number { get; set; }
  public bool Paid { get; set; }
}

V3008. The 'Amount' variable is assigned values twice successively. Perhaps this is a mistake. Check lines: 148, 142. BillingInfo.cs 148

At the time of writing the article, we could only guess what exactly the developers wanted. Either the first assignment is redundant, or vise versa — the second assignment is superfluous. However, after the article was published, we contacted the developers and reported the issues we found. The developers fixed them — in this case, they just removed the first assignment.

8th place. An element from the void in Barotrauma

The 8th place goes to the error we described in the article about the Barotrauma check:

public ParticlePrefab(XElement element, ContentFile file)
{
  ....
  if (CollisionRadius <= 0.0f) 
    CollisionRadius = Sprites.Count > 0 ? 1 : 
                                       Sprites[0].SourceRect.Width / 2.0f;
}

V3106 Possibly index is out of bound. The '0' index is pointing beyond 'Sprites' bound. ParticlePrefab.cs 303

Accessing the first element of an empty collection is clearly something strange :). We can only hope that the exception thrown doesn't affect the gameplay.

7th place. Initializing fields in AvalonStudio

The seventh place goes to well-hidden error we found when checking AvalonStudio:

public class ColorScheme
{
  private static List<ColorScheme> s_colorSchemes =
    new List<ColorScheme>();
  private static Dictionary<string, ColorScheme> s_colorSchemeIDs =
    new Dictionary<string, ColorScheme>();
  private static readonly ColorScheme DefaultColorScheme = 
    ColorScheme.SolarizedLight;
  ....
  public static readonly ColorScheme SolarizedLight = new ColorScheme
  {
    ....
  };
}

Oh, seems like this time I won't show the analyzer warnings right away :) Just so you can feel what it's like to search for such issues without static analysis. This error is simple though, so I'm sure you'll handle it.

What's the matter here?

The analyzer issues the following warning:

V3070. Uninitialized variable 'SolarizedLight' is used when initializing the 'DefaultColorScheme' variable. ColorScheme.cs 32

Indeed, there's something wrong with the field initialization.

6th place. Confusion in .NET 7

Yes, we found this error in the famous .NET 7 — here's an article about other errors we found there. Nobody's perfect :).

First, let's look at the* XmlConfigurationElementTextContent* constructor:

public XmlConfigurationElementTextContent(string textContent, 
                                          int? linePosition, 
                                          int? lineNumber)
{ .... }

Now let's see where it's used:

public static IDictionary<string, string?> Read(....)
{
  ....
  case XmlNodeType.EndElement:
    ....
    var lineInfo = reader as IXmlLineInfo;
    var lineNumber = lineInfo?.LineNumber;
    var linePosition = lineInfo?.LinePosition;
    parent.TextContent =
      new XmlConfigurationElementTextContent(string.Empty, 
                                             lineNumber,
                                             linePosition);
    ....
    break;
  ....
  case XmlNodeType.Text:
    ....
    var lineInfo = reader as IXmlLineInfo;
    var lineNumber = lineInfo?.LineNumber;
    var linePosition = lineInfo?.LinePosition;

    XmlConfigurationElement parent = currentPath.Peek();

    parent.TextContent =
      new XmlConfigurationElementTextContent(reader.Value,
                                             lineNumber,
                                             linePosition);
    ....
    break;
  ....
}

Pay attention to the order of arguments and parameters:

arguments: ..., lineNumber, linePosition;
parameters: ..., linePosition, lineNumber.

That's exactly what the analyzer detected:

V3066 Possible incorrect order of arguments passed to 'XmlConfigurationElementTextContent' constructor: 'lineNumber' and 'linePosition'. XmlStreamConfigurationProvider.cs 133
V3066 Possible incorrect order of arguments passed to 'XmlConfigurationElementTextContent' constructor: 'lineNumber' and 'linePosition'. XmlStreamConfigurationProvider.cs 148

My teammate created an issue on GitHub: the developers fixed the order of arguments and added a test.

5th place. Orleans: don't rush with clearing

The error from the Orleans project check opens the second half of our top:

private class BatchOperation
{
  private readonly List<TableTransactionAction> batchOperation;
  ....

  public async Task Flush()
  {
    if (batchOperation.Count > 0)
    {
      try
      {
        ....
        batchOperation.Clear();                              // <=
        keyIndex = -1;

        if (logger.IsEnabled(LogLevel.Trace))
        {
          for (int i = 0; i < batchOperation.Count; i++)     // <=
          {
            logger.LogTrace(....)
          }
        }
      }
      catch (Exception ex)
      {
        ....
      }
    }
  }
}

V3116. Consider inspecting the 'for' operator. It's possible that the loop will be executed incorrectly or won't be executed at all. AzureTableTransactionalStateStorage.cs 345

Something tells me that the developers should have cleared batchOperation a bit later. Judging by the fix, the Orleans developers agree with me.

4th place. Shadow features of Eto.Forms

An error described in the article about Eto.Forms, the GUI framework almost made it to the top 3 errors:

public NSShadow TextHighlightShadow
{
  get
  {
    if (textHighlightShadow == null)
    {
      textHighlightShadow = new NSShadow();
      textHighlightShadow.ShadowColor = NSColor.FromDeviceWhite(0F, 0.5F);
      textHighlightShadow.ShadowOffset = new CGSize(0F, -1.0F);
      textHighlightShadow.ShadowBlurRadius = 2F;
    }
    return textHighlightShadow;
  }
  set { textShadow = value; }         // <=
}

public NSShadow TextShadow
{
  get
  {
    if (textShadow == null)
    {
      textShadow = new NSShadow();
      textShadow.ShadowColor = NSColor.FromDeviceWhite(1F, 0.5F);
      textShadow.ShadowOffset = new CGSize(0F, -1.0F);
      textShadow.ShadowBlurRadius = 0F;
    }
    return textShadow;
  }
  set { textShadow = value; }
}

PVS-Studio warning: V3140 Property accessors use different backing fields. Eto.Mac64 MacImageAndTextCell.cs 162

Copy-paste says hello :).

Perhaps, assigning a value to the TextHighlightShadow property should somehow affect textShadow. But even so, it's not clear why it doesn't affect the textHighlightShadow field used in the get accessor.

3d place. Lost migration in Squidex

The bronze goes to an error we described in the article about checking the ASP.NET Core projects. This error was found in Squidex:

private IEnumerable<IMigration?> ResolveMigrators(int version)
{
  yield return serviceProvider.GetRequiredService<StopEventConsumers>();

  // Version 06: Convert Event store. Must always be executed first.
  if (version < 6)
  {
    yield return serviceProvider.GetRequiredService<ConvertEventStore>();
  }

  // Version 22: Integrate Domain Id.
  if (version < 22)
  {
    yield return serviceProvider
                   .GetRequiredService<AddAppIdToEventStream>();
  }

  // Version 07: Introduces AppId for backups.
  else if (version < 7) 
  {
    yield return serviceProvider
                   .GetRequiredService<ConvertEventStoreAppId>();
  }

  // Version 05: Fixes the broken command architecture and requires a
  // rebuild of all snapshots.
  if (version < 5)
  {
    yield return serviceProvider.GetRequiredService<RebuildSnapshots>();
  }
  else
  {
    // Version 09: Grain indexes.
    if (version < 9)
    {
      yield return serviceProvider.GetService<ConvertOldSnapshotStores>();
    }

    ....
  }

  // Version 13: Json refactoring
  if (version < 13)
  {
    yield return serviceProvider
                   .GetRequiredService<ConvertRuleEventsJson>();
  }

  yield return serviceProvider.GetRequiredService<StartEventConsumers>();
}

Good luck finding the error in the code :).

Or you can look at the shortened fragment and try to find it again:

// Version 22: Integrate Domain Id.
if (version < 22)
{
  yield return serviceProvider.GetRequiredService<AddAppIdToEventStream>();
}

// Version 07: Introduces AppId for backups.
else if (version < 7) 
{
  yield return serviceProvider
                 .GetRequiredService<ConvertEventStoreAppId>();
}

I'm sure it's way easier to search for it now. Oh, if only there were a button that would show only the code fragments related to errors... Wait, this button exists!

Clicking on this miracle button helped discover this issue:

V3022. Expression 'version < 7' is always false. MigrationPath.cs 55

Indeed, the version < 7 condition is checked only if version >= 22. Strange, isn't it?

My IDE doesn't have that button

If you want to get this button, download the extension here.

If your IDE is not supported, you can contact us and express your interest in the plugin with the button for it.

2nd place. Funny shifts in Discord.NET

This time the silver goes to an error we found when checking the Discord.NET project:

public enum GuildFeature : long
{
  None = 0,
  AnimatedBanner = 1 << 0,
  AnimatedIcon = 1 << 1,
  Banner = 1 << 2,
  ....
  TextInVoiceEnabled = 1 << 32,
  ThreadsEnabled = 1 << 33,
  ThreadsEnabledTesting = 1 << 34,
  ....
  VIPRegions = 1 << 40,
  WelcomeScreenEnabled = 1 << 41,
}

Just like the previous case, I'll ~~have a pleasure to~~ let you find this error first. Or you can click on the miracle button:

V3134. Shift by 32 bits is greater than the size of 'Int32' type of expression '1'. GuildFeature.cs 147

The problem is, the type of shift operands here is int, so, the result will have the same type. In this case, a value of the 1 << 32 type is 1, the value of 1 << 33 is the same as 1 << 1, and so on.

So, it leads to a weird situation: the AnimatedBanner and TextInVoiceEnabled elements are the same — just like the AnimatedIcon-ThreadsEnabled pair, etc. Besides, the internal type of enumeration is long, and now it becomes clear that the developer did not expect this behavior at all.

1st place. Vulnerability from BlogEngine.NET

And we finally get to the most interesting bug described in 2022. Please welcome — the XXE vulnerability from BlogEngine.NET:

public XMLRPCRequest(HttpContext input)
{
  var inputXml = ParseRequest(input);

  // LogMetaWeblogCall(inputXml);
  this.LoadXmlRequest(inputXml); // Loads Method Call 
                                 // and Associated Variables
}

private static string ParseRequest(HttpContext context)
{
  var buffer = new byte[context.Request.InputStream.Length];

  context.Request.InputStream.Position = 0;
  context.Request.InputStream.Read(buffer, 0, buffer.Length);

  return Encoding.UTF8.GetString(buffer);
}

private void LoadXmlRequest(string xml)
{
  var request = new XmlDocument();
  try
  {
    if (!(xml.StartsWith("<?xml") || xml.StartsWith("<method")))
    {
      xml = xml.Substring(xml.IndexOf("<?xml"));
    }

    request.LoadXml(xml);              // <=
  }
  catch (Exception ex)
  {
    throw new MetaWeblogException("01", 
                     $"Invalid XMLRPC Request. ({ex.Message})");
  }
  ....
}

Wait. What? What vulnerability?

Even though I showed you only those methods that relate to the error, it's still hard to see a vulnerability to an XXE attack.

Read the article to find a detailed description of such vulnerabilities in general and this one in BlogEngine.NET in particular. Here, I'll briefly describe the most important parts.

First, look at the ParseRequest method:

private static string ParseRequest(HttpContext context)
{
  var buffer = new byte[context.Request.InputStream.Length];

  context.Request.InputStream.Position = 0;
  context.Request.InputStream.Read(buffer, 0, buffer.Length);

  return Encoding.UTF8.GetString(buffer);
}

context.Request.InputStream contains some data sent by a user to an apllication. If the data is not checked, then InputStream may contain anything. Such data is called potentially tainted (read more here).

The ParseRequest method reads the request data into a buffer, converts it into a string and then returns it "as is". Where does the data go?

The answer is here:

public XMLRPCRequest(HttpContext input)
{
  var inputXml = ParseRequest(input);

  // LogMetaWeblogCall(inputXml);
  this.LoadXmlRequest(inputXml); // Loads Method Call 
                                 // and Associated Variables
}

We see that the result of the ParseRequest work is passed to the LoadXmlRequest method. Once more, the data may contain anything, since only the user controls its contents. Now let's look at how the data is processed in the LoadXmlRequest method:

private void LoadXmlRequest(string xml)
{
  var request = new XmlDocument();
  try
  {
    if (!(xml.StartsWith("<?xml") || xml.StartsWith("<method")))
    {
      xml = xml.Substring(xml.IndexOf("<?xml"));
    }

    request.LoadXml(xml);              // <=
  }
  catch (Exception ex)
  {
    throw new MetaWeblogException("01", 
                     $"Invalid XMLRPC Request. ({ex.Message})");
  }
  ....
}

That's where the vulnerability shows itself! The potentially tainted data is first passed to the xml parameter and then — to the LoadXml method of the XmlDocument class.

You may think: "Well, and? We have try-catch here, and even a check". The problem is, they won't save you from the real threat.

If an attacker passes certain data to the request, then the contents of almost any file of a system, in which the request is processed, will be written to the request variable.

The article describes how the contents may return to the user. It shows not just a "theoretical danger", but a completely working scenario for exploiting the vulnerability.

How is it that processing custom xml leads to reading files from the system? A brief explanation by the analyzer:

V5614. Potential XXE vulnerability inside method. Insecure XML parser is used to process potentially tainted data from the first argument: 'inputXml'. BlogEngine.Core XMLRPCRequest.cs 41

Thus, PVS-Studio detects and points out 2 components of the XML External Entity vulnerability:

unsafe (unsafely configured) XML parser;
tainted data passed to this parser.

We have already mentioned the tainted data. However, it can be difficult to understand how the parser may be unsafe. In this example, the default configuration used is unsafe. However, in general, everything is more complicated. You can read more about this vulnerability and ways to protect from it in the article: "Vulnerabilities due to XML files processing: XXE in C# applications in theory and in practice".

The error in BlogEngine.NET is completely different from others discussed in this article — it is not just "strange code", but a real vulnerability. There is an entry for it in the CVE database.

Conclusion

In 2022 we published many articles, thanks to which I managed to collect a diverse top. It also allows you to evaluate the variety of real errors that PVS-Studio can find. Some of them are the result of writing code in a rush, some are the result of not fully understanding the language features. Some of the errors hide until the analyzer finds them.

As usual, I'll leave a link to the page where you can download PVS-Studio and evaluate it on your project. That's all folks!

Collecting the best C++ practices

Unicorn Developer — Fri, 13 Jan 2023 08:04:25 +0000

Reading our articles, you may wonder: why are they always talking about bad things? This code contains bug; there are code smells here; it's an antipattern — don't code like that. So, would you help me to look on the bright side of C++ programming? Feel free to leave comments 💬

Recently I came across a discussion on Reddit "Good repos for beginners to browse that follow best modern C++ practices (including testing, static analysis etc...)". The thread is not the longest one, but it's still quite interesting. Then, an idea popped into my head — why don't I post a survey on Habr.com? I did the same with terrible coding tips.

By the way, the idea to write an article about terrible coding tips is evolving. Soon I plan to publish a book with 60 anti-patterns in C++. Subscribe to our monthly newsletter, if you don't want to miss the most interesting publications.

So, the topic of terrible coding tips is covered :). It's time to talk about good practices!

I go first. Then, I invite you to join me and write in the comments:

what tools you recommend;
what interesting and useful libraries you use;
what projects are useful in learning C++;
what practices, coding standards you can advise;
and so on.

Starters

ModernCppStarter. Kick-start your C++! A template for modern C++ projects using CMake, CI, code coverage, clang-format, reproducible dependency management and much more.
gui_starter_template. This is a C++ Best Practices GitHub template for getting up and running with C++ quickly.

Projects to learn C++

Note to self. If I write an article on the collected information later, I need to check these and other proposed projects with PVS-Studio. Libraries are different, you know... I don't want to recommend something potentially buggy as an example to follow.

Diligent Engine. A Modern Cross-Platform Low-Level 3D Graphics Library and Rendering Framework Tweet.
JSON for Modern C++. Intuitive syntax. Trivial integration. Serious testing. Memory efficiency. Speed.
Stroika is a modern, portable, thread-savvy, C++ application framework. It makes writing high performance C++ applications easier by providing safe, flexible, modular building blocks.
concurrencpp. Modern concurrency for C++. Tasks, executors, timers and C++20 coroutines to rule them all.
awesome-hpp. A curated list of awesome header-only C++ libraries.

Speeding up the build

Here I invite you to read my colleague's article "Speeding up the build of C and C++ projects".
There are various old and new discussions on this topic. I am excited to learn about new, interesting things on this topic from you. Thanks in advance.

Coding standards and style guides

The C++ Core Guidelines are a collaborative effort led by Bjarne Stroustrup, much like the C++ language itself. They are the result of many person-years of discussion and design across a number of organizations.
Collaborative Collection of C++ Best Practices. This online resource is part of Jason Turner's collection of C++ Best Practices resources. By the way, since I mentioned Jason, here's the link to his C++ Weakly channel.
Google C++ Style Guide. The goal of this guide is to manage this complexity by describing in detail the dos and don'ts of writing C++ code. These rules exist to keep the code base manageable while still allowing coders to use C++ language features productively.

Dynamic code analysis

AddressSanitizer finds memory errors.
LeakSanitizer searches for memory leaks.
ThreadSanitizer detects data races and deadlocks.
MemorySanitizer looks for uninitialized memory.
HWASAN, or Hardware-assisted AddressSanitizer, a new variant of AddressSanitizer that uses less memory.
UBSan finds undefined behavior in a program.

Static code analyzers

Here I will play dumb and mention only our PVS-Studio. Well, why not? It's a wonderful and powerful static code analyzer. The tool helps find lots of errors and potential vulnerabilities even at the stage of writing C++ code.
TODO. I'm also excited to hear about your successful experiences of using other static code analyzers.

Books and other references

I always recommend everyone to read Code Complete by Steve McConnell (ISBN 0-7356-1967-0).
C++ Best Practices (2019).
The Ultimate Question of Programming, Refactoring, and Everything.
C++ Coding Standards: 101 Rules, Guidelines, and Best Practices.
Scott Meyers. Effective C++: 55 Specific Ways to Improve Your Programs and Designs (3rd edition).
Scott Meyers. Effective Modern C++: 42 Specific Ways to Improve Your Use of C++11 and C++14".

Now, it's your turn 💥

Please share everything that you consider useful to C++ programmers!

How has LINQ performance enhanced in .NET 7?

Unicorn Developer — Wed, 11 Jan 2023 12:18:29 +0000

How has LINQ performance enhanced in .NET 7?

New version of .NET enhanced the performance of the Min, Max, Average and Sum methods for arrays and lists. How much do you think their execution speed has increased? 2x or 5x? No, they got even faster. Let's see how that was achieved.

How has LINQ enhanced?

LINQ (Language-Integrated Query) is a simple and convenient query language. It allows you to express complex operations in a simple way. Almost every .NET developer uses LINQ. However, this simplicity of use comes at a price of the execution speed and extra memory allocation. In most situations, it has no significant effect. However, in cases when performance is critical, these limitations may be pretty unpleasant.

So, the recent update has enhanced the performance of the following methods: Enumerable.Max, Enumerable.Min, Enumerable.Average, Enumerable.Sum.

Let's see how their performance increased using the following benchmark:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Collections.Generic;
using System.Linq;


[MemoryDiagnoser(displayGenColumns: false)]
public partial class Program
{
  static void Main(string[] args) =>
    BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

  [Params (10, 10000)]
  public int Size { get; set; }
  private IEnumerable<int> items;

  [GlobalSetup]
  public void Setup()
  {
    items = Enumerable.Range(1, Size).ToArray();
  }  

  [Benchmark]
  public int Min() => items.Min();

  [Benchmark]
  public int Max() => items.Max();

  [Benchmark]
  public double Average() => items.Average();

  [Benchmark]
  public int Sum() => items.Sum();
}

Benchmark results:

Method	Runtime	Size	Mean	Ratio	Allocated
Min	.NET 6.0	10	75.491 ns	1.00	32 B
Min	.NET 7.0	10	7.749 ns	0.10	-

Max	.NET 6.0	10	71.128 ns	1.00	32 B
Max	.NET 7.0	10	6.493 ns	0.09	-

Average	.NET 6.0	10	68.963 ns	1.00	32 B
Average	.NET 7.0	10	7.315 ns	0.11	-

Sum	.NET 6.0	10	69.509 ns	1.00	32 B
Sum	.NET 7.0	10	9.058 ns	0.13	-

Min	.NET 6.0	10000	61,567.392 ns	1.00	32 B
Min	.NET 7.0	10000	2,967.947 ns	0.05	-

Max	.NET 6.0	10000	56,106.592 ns	1.00	32 B
Max	.NET 7.0	10000	2,948.302 ns	0.05	-

Average	.NET 6.0	10000	52,803.907 ns	1.00	32 B
Average	.NET 7.0	10000	2,967.810 ns	0.06	-

Sum	.NET 6.0	10000	52,732.121 ns	1.00	32 B
Sum	.NET 7.0	10000	5,897.220 ns	0.11	-

The results show that the execution time for finding the minimum element of an array has generally decreased. In 10 times for small arrays and 20 times for arrays containing 10,000 elements. Similarly, for other methods (except for finding the sum, the difference between the sizes of the collections did not affect the results much).

It is also worth noting that in .NET 7 no additional memory is allocated when methods are called.

Let's check how these methods work with List.

Method	Runtime	Size	Mean	Ratio	Allocated
Min	.NET 6.0	10	122.554 ns	1.00	40 B
Min	.NET 7.0	10	8.995 ns	0.07	-

Max	.NET 6.0	10	115.135 ns	1.00	40 B
Max	.NET 7.0	10	9.171 ns	0.08	-

Average	.NET 6.0	10	110.825 ns	1.00	40 B
Average	.NET 7.0	10	8.163 ns	0.07	-

Sum	.NET 6.0	10	113.812 ns	1.00	40 B
Sum	.NET 7.0	10	13.197 ns	0.12	-

Min	.NET 6.0	10000	91,529.841 ns	1.00	40 B
Min	.NET 7.0	10000	2,941.226 ns	0.03	-

Max	.NET 6.0	10000	84,565.787 ns	1.00	40 B
Max	.NET 7.0	10000	2,957.451 ns	0.03	-

Average	.NET 6.0	10000	81,205.103 ns	1.00	40 B
Average	.NET 7.0	10000	2,959.882 ns	0.04	-

Sum	.NET 6.0	10000	81,857.576 ns	1.00	40 B
Sum	.NET 7.0	10000	5,783.370 ns	0.07	-

In .NET 6, all operations on arrays much faster than on lists. The same is true for small collections in .NET 7. As the number of elements increases, lists are equal to arrays in performance.

According to the test results, lists' performance increased by 31 times.

But how could this be achieved?

Let's take a closer look at the implementation of the Min method.

That's how the Min method is implemented in .NET 6:

public static int Min(this IEnumerable<int> source)
{
  if (source == null)
  {
    ThrowHelper.ThrowArgumentNullException(ExceptionArgument.source);
  }

  int value;
  using (IEnumerator<int> e = source.GetEnumerator())
  {
    if (!e.MoveNext())
    {
      ThrowHelper.ThrowNoElementsException();
    }

    value = e.Current;
    while (e.MoveNext())
    {
      int x = e.Current;
      if (x < value)
      {
        value = x;
      }
    }
  }
  return value;
}

The method is quite simple. We get the IEnumerable collection, take the collection element and use the MoveNext method to get the next element. We compare them, save the one that is smaller, and repeat until the end of the collection.

The new version of the Min method is different:

public static int Min(this IEnumerable<int> source) => MinInteger(source);

The MinInteger method is applied to a collection of integers. Let's examine it in more details.

private static T MinInteger<T>(this IEnumerable<T> source)
  where T : struct, IBinaryInteger<T>
{
  T value;

  if (source.TryGetSpan(out ReadOnlySpan<T> span))
  {
    if (Vector.IsHardwareAccelerated && 
        span.Length >= Vector<T>.Count * 2)
    {
      .... // Optimized implementation
      return ....;
    }
  }
  .... //Implementation as in .NET 6
}

Primarily, we try to get the object of the ReadOnlySpan type from the provided collection. If we fail to get the ReadOnlySpan collection representation, then the code branch (that matches the implementation of the Min method in .NET 6) is executed. In our case, we can get ReadOnlySpan, since we use arrays and lists.

But what is ReadOnlySpan actually? The Span and ReadOnlySpan types provide safe representation of a continuous managed and unmanaged memory area. The structure of the Span type is defined as a ref struct. This means that it can only be placed on the stack, which makes it possible to avoid allocating additional memory and improves data performance.

The* Span* type has also undergone some changes in the new version of C#. Since C# 11 introduced the option to create reference fields in a ref struct, the internal representation of Span has changed. Previously, a special internal type — ByReference, was used to store a reference to the beginning of the memory area, but there was no security check in it. Now ref fields are used for this purpose. It provides a more secure memory operation. You can read more about other changes in C# 11 in the article

But let's get back to the Min method. When you get ReadOnlySpan, the method tries to perform a vector search using the Vector type. To do this, the following condition should be met:

if (Vector.IsHardwareAccelerated && span.Length >= Vector<T>.Count * 2)

The first part of the condition checks whether the Vector.IsHardwareAccelerated property returns true. Let's take a look at the implementation of this property.

public static bool IsHardwareAccelerated
{
  [Intrinsic]
  get => false;
}

The [Intrinsic] attribute is applied to the getter. The attribute indicates that the value returned by IsHardwareAccelerated can be replaced JIT. The property returns true if hardware acceleration can be applied to operations on vectors through built-in JIT support, otherwise false is returned. To enable hardware acceleration, you need to run the build for the x64 platform with the Release configuration or build the project for AnyCPU with the "Prefer 32-bit" setting disabled.

To fulfill the second part of the condition, the size of Span should be at least 2 times the size of the vector.

How is the size of this vector calculated?

The Vector data type is used in the new implementation of the Min method to create a vector. This type is a wrapper for three other types: Vector64, Vector128 and Vector256. They contain vectorized data of the corresponding length. Vector128 can store 16 8-bit, 8 16-bit, 4 32-bit or 2 64-bit values. The type to be used is selected depending on whether it is supported by the processor or not.

Thus, if the Vector128 type is used when executing the method, then the Span type obtained from the array or list must contain 8 or more elements of the int type.

If all conditions are met, the method uses the advantages of the ReadOnlySpan and Vector types to optimize finding the minimum:

private static T MinInteger<T>(this IEnumerable<T> source)
where T : struct, IBinaryInteger<T>
{
  .... 
  if (Vector.IsHardwareAccelerated && span.Length >= Vector<T>.Count * 2)
  {
    var mins = new Vector<T>(span);
    index = Vector<T>.Count;
    do
    {
      mins = Vector.Min(mins, new Vector<T>(span.Slice(index)));
      index += Vector<T>.Count;
    }
    while (index + Vector<T>.Count <= span.Length);

    value = mins[0];
    for (int i = 1; i < Vector<T>.Count; i++)
    {  
      if (mins[i] < value)
      {
        value = mins[i];
      }
    }
  ....
}

Let's explore how the vectorized implementation of finding a minimum works. A vector is created from the first N Span elements (N depends on the type of vector; for Vector128 it is 4 elements). This vector is compared with a vector of the following N elements in Vector.Min. The resulting vector contains the minimum value of each pair of elements of the two given vectors. Then the next part of Span is taken, and the search continues. You only need to find the minimum value among those stored in the resulting vector. The advantage of using a vector is that all operations on its elements occur simultaneously.

An example of using Span and vectorization to optimize the Min method is shown above. But what about the other methods? You can meet similar features for the Max, Average, Sum methods. The optimized versions of these methods are available for arrays and lists of int, long, float, double and decimal types.

Conclusion

Microsoft developers used Span and hardware acceleration to work with vectors. As a result, the .NET community brought visible enhancement to LINQ performance. Now we can use advanced methods in cases where the speed of code execution, and allocated memory are critical.

Actually, .NET 7 got a few other performance improvements. You can read about them in .NET Blog.

Christmas holidays with PVS-Studio

Unicorn Developer — Fri, 23 Dec 2022 13:03:00 +0000

Christmas holidays are the most beloved and cherished ones all over the world. Gifts, decorations, family and friends all gathered together — these are the essentials of Christmas and New Year's buzz. Our team was also caught up in this. In the eve of the Christmas holidays, we have prepared various treats and gifts for you. Continue reading to find out which ones!

The quiz: Save Christmas from bugs

Based on the feedback we got after releasing previous quizzes, we can safely say that our audience likes this format. That's why we decided to make a Christmas special for you.

When there are Christmas lovers, there are also Christmas haters. And they'd love to spoil all these celebrations. In our case, these haters are bugs. We prepared 8 stories for you — each of them tells about bugs that plotted against well-known companies. Help us find bugs disguised in these cases, figure out what they did and what havoc they caused. Christmas mishaps, treacherous bugs, terrifying (and not) consequences, and, of course, gifts — they are waiting for you in our Christmas quiz. Well, are you ready? Let's save the holidays!

The game: Endless coding

Have you ever thought that your work day is like a video game? Deadlines, coffee breaks, bugs — you can find them all in "Endless coding".

Avoid obstacles like technical debt, bugs, burning deadlines, etc. Catch buffs in bubbles — cups of coffee, donuts, pay raise and promotion, or help from PVS-Studio. Looks just like an ordinary work day, doesn't it?

Take the highest place in the leaderboard and prove your team leader that you're not playing games here😊!

Players who score the highest points will get a nice bonus in the end!

New Year's survey

Time doesn't wait for anyone — 2023 came in the blink of an eye. In order to keep up with the times, the PVS-Studio team decided to get to know our newcomers as well as have a chat with our regulars. We want to get better for our users and we want to know what matters the most for you. So, tell us about this — take a survey. We appreciate any feedback.

A gift will be waiting for you at the end of it.

Thank you for being with us all these years. After all, we enhance and develop our tool for you.

Happy holidays! See you in 2023! ✨

Save Christmas from Bug!

Unicorn Developer — Wed, 21 Dec 2022 08:02:17 +0000

Anyone in Bug's shoes would turn green with envy and go mad. As a true introvert, he lives in the depths of the code and hides far away from developers, reviewers, testers, and everyone else 🐞 But these "everyone else" are preparing the biggest Christmas party ever 🎄🎅🎁 They make a fuss over the holidays, decorate everything they can decorate and make Bug so angry! He will not resist the urge of ruining their holidays. Bug decides to steal Christmas 😱

Don't let Bug ruin the holidays! Take the quiz to save Christmas from the offended winged beast 🦸

A software bug captured Apple and other huge companies

Unicorn Developer — Wed, 07 Dec 2022 14:59:05 +0000

We collected some hot stories about programming errors for you to have a little fun and learn something new. Keep reading to find out how one programmer broke the Internet by deleting a tiny piece of code and see other not-so-obvious errors.

"Error" is something that can scare even experienced developers. Everyone faces them at some point. Today you'll see that everyone may screw up, and mistakes shouldn't frustrate you. We have already published related articles — Stories about Christmas and New Year Bugs and others are listed in the end.

Who wants to be a millionaire: the Chilean version

This story is about the error that turned the Chilean man's life around. In May, 2022, a man in Chile was accidentally paid 165 million pesos (£150,000), 330 times his salary. This happened due a payroll blunder error.

The worker reported about the error to the deputy manager. The manager asked the worker to go to his bank and initiate the return of the excess money. You could think the man got back the money, and the story was over. But it wasn't.

Well, the man promised his employer to show up in the bank and get the amount refunded. However, a day later, he was off the radar. After two days, he appeared and promised to retrieve the amount soon — probably, he was still on the fence :). Diario Financiero reported that, the worker later sent a resignation letter via his attorney, and dropped out of the sight.

The company he was employed has now reached out to various agencies in attempt to get back the money and registered a complaint of misappropriating funds against the employee. Let's hope this story keeps going.

The £50 million electricity bill

A couple from Lancashire were shocked when their monthly electricity bill had increased from £87 to £53,480,062. "Unfortunately, I don't have this kind of money in my account. It was a massive shock," Mr. Brotherton, a 63-year-old accountant, said.

Nigel and Linda Brotherton had a new electricity meter installed, but an electrician didn't connect the wires correctly. This led to a wrong measure of the electricity usage. Due to this mistake, their power supplier calculated the bill and sent it to their bank account.

"My wife was worried if they had tried to take the money it could have affected our credit rating," Mr. Brotherton said. After the recovering from initial shock, the couple contacted the power supplier, who apologized and promised to correct the wires.

iPhone's first glitches a month after release

The accident took place a week after iPhone 14 Pro was released. Some owners have reported that their iPhones cause camera shaking when they use TikTok, Instagram, and Snapchat.

Multiple users tweeted the videos of how it looked like — opening the camera in certain apps causes it to shake and produce audible grinding sounds. That's quite a creepy thing. However, if you use the built-in camera, nothing is wrong.

Apple reported that the problem was caused by the software issue and not by the third-party applications. Apple spokesperson Alex Kirschner stated in his email to The Verge, that the company is "aware of the issue and a fix will be released next week". However, Apple didn't explain what exactly had caused the error, and why some users encountered this problem and others did not.

Apple introduced a new feature that could actually help you in car accidents — if your iPhone 14 detects a severe car crash, it can help you contact emergency services and notify your emergency contacts. That's impressive. But what if an accident takes place on rollercoasters? "You're raving, dude", — you think. Well, the iPhone 14 is sure it's possible.

The car accident detection feature has set off false alarms for emergency personnel near Cincinnati's Kings Island amusement park. The same crash detection technology is also featured on the Apple Watch 8. Officials say the fix is simple: Putting the iPhone 14 and the Apple Watch 8 on airplane mode before boarding a rollercoaster. So, here is the burning question — should we expect signs in the amusement park like "turn off your iPhones before you get on the rollercoaster?"

Let's hope that there are only two glitches in the latest iPhone model and there would be nothing that could dishonor the company's impeccable reputation.

How the Windows developer stole 2022 year

On 2021 New Year's Eve, the Windows team decided to wish their Twitter followers a Happy New Year. The official Windows Developer account tweeted the following:

That's C# console code. If you run this app when it's exactly midnight on New Year's Eve, you'll get a "Happy New Year!" message. But if you run it at any other time, it will show you that it's still 2021 — that's how this app should work. But it didn't deliver the expected result.

There is a problem — you can get the New Year greeting at exactly midnight. I mean you will fail to get the greeting if it is one second into 2022 or one second before 2022. Moreover, if it's 2022 now, the app will tell you that it's still 2021 — no matter what the date or time is. It works only for midnight...

The Windows developer realized the error and deleted the tweet. You can watch Scott Hanselman's overly-detailed analysis of the many problems with this code. If you're an early-career developer, you might find this interesting.

The "left-pad" havoc or how a programmer broke the Internet by deleting a tiny piece of code

In March, 2016, Azer Koçulu, a programmer in Oakland, California broke the Internet around the world by deleting 11 lines of code. Koçulu published his code on npm, a popular tool among JavaScript developers that is used to find and install open-source software.

Koçulu wrote the project and called it "kik". This project was intended to help programmers set up templates for their projects. By chance, the messaging app created in Ontario, Canada, shares the name with the Koçulu's project. On March 11, Koçulu received an email from Bob Stratton, a trademark agent of Kik. Bob Stratton asked Koçulu to change the project's name for legal reasons. Koçulu rejected his suggestion. Here you can find the whole discussion (and find out why two parts failed to reach an agreement). After that, Koçulu intended to take down the packages he had registered on npm. "I think I have the right of deleting all my stuff from NPM" was the email, that ended up the dialogue between Koçulu and Bob Stratton.

After a few days, JavaScript programmers around the world started receiving a strange error message when they tried to run their code:

Npm ERR! 404 'left-pad' is not in the npm registry.

The point is, the code they were trying to run required a package called left-pad, but the npm registry didn't have it. After that, developers were turned to the GitHub repository, where the left-pad was supported. The code that they found looks as follows:

module.exports = leftpad;
function leftpad (str, len, ch)  {
str = String(str);
var i = -1;
if (!ch && ch !== 0)
ch = ' ';
len = len - str.length;
while (++i < len)  {
str = ch + str;
}
Return str;
}

There are only eleven code lines. This is a single-purpose function, simple enough for most developers to write themselves, but lots of npm packages relied on left-pad to do it for them, which is how this tiny bit of code became so important, but unknown.

Some of the largest, most widely used npm packages platforms have suddenly been broken. React, a JavaScript library for building user interfaces was one of the affected packages. And many smaller websites such as Quartz's own Atlas. Its absence has been felt around the world — developers from Australia, Germany, the United States, and the Czech Republic commented on the left-pad page on GitHub.

In Ontario, where the problem occurred, Kik's developers had left-pad problems as well. Two hours later, they added these 11 lines of code to the npm registry. This situation proved to be a valuable lesson for developers. It helped developers understand that wide interests are more important than the wishes of one author.

Conclusion

I hope that this article set you thinking that mistakes are no big deal. If you are still afraid of programming errors, there are various tools to protect yourself. For example, our company — PVS-Studio — develops the static analyzer that can easily detect even ambiguous programming errors. By the way, we have collected various errors found in open-source projects as well. There are Chromium, Linux core, Tizen OS and even Windows Calculator projects listed, you may find it interesting to look at someone else's mistakes :).

Let me end up with a phrase from The Simpsons episode, where Lenny says: "Everyone makes mistakes, that's why pencils have erasers".

PS: didn't the errors seem funny enough to you? Then here is a collection of hilarious memes about software errors that people have encountered in their daily lives:

1. How?

2. What the hell did I do

3. Well, great...

4. Scuber Diver

5. This is getting ridiculous

6. I don't really know what to say

7. Homonyms are such a bliss to a Virtual Assistant

8. Oh no, my PC ran away

The pictures were originally published on boredpanda.com.

Additional links

Catastrophic backtracking: how can a regular expression cause a ReDoS vulnerability?

Unicorn Developer — Wed, 30 Nov 2022 14:47:54 +0000

Regular expressions come in handy when you need to search for and replace text. However, in some cases, they may cause the system to slow down or even make vulnerable to ReDoS attacks.

Introduction

ReDoS is a subtype of a DoS attack. The aim of a ReDoS attack is to halt an application or cause it to slow down via an inefficient regex.

ReDoS attacks can be divided into two types:

A string with a malicious pattern is passed to an application. Then this string is used as a regex, which leads to ReDoS.
A string of a certain format is passed to an application. Then this string is evaluated by a vulnerable regex, which leads to ReDoS.

The main point of any ReDoS attack is using a vulnerable regular expression in an application. Passing a string of a certain format to a regex leads to its unreasonably long calculation.

If a ReDoS attack is successful, then the regex calculation results in catastrophic backtracking. It is a consequence of the backtracking function in the Regex engine, which iterates through possible string matches until it finds the correct one. If there's no correct match, a regular expression won't stop until it iterates through all possible options. A complete iteration of all possible options leads to an unacceptably long calculation of a regex. This is called catastrophic backtracking.

A regex is vulnerable to catastrophic backtracking if it contains at least one subexpression that may cause a large number of matching options.

Catastrophic backtracking: real examples

Let's inspect several regular expressions for vulnerabilities.

I wrote a small program — it displays a graph of how the calculation time of regex depends on the number of characters in the evaluated string. In next examples, I will use this program to show you the catastrophic backtracking.

Example 1

Let's look at a simple synthetic example:

(x+)+y

Let's compare the calculation time of the (x+)+y expression in two cases:

The input to the regular expression accepts strings corresponding to the specified pattern one by one. At the same time the length of each subsequent string is more than the previous one by 1.
The input to the regular expression accepts strings that do not match the pattern (there is no y character at the end of the string). At the same time the length of each subsequent string is more than the previous one by 1.

The results of such an experiment are below:

Figure 1. The execution time of a regular expression if a string matches the pattern (x+)+y.

Figure 2. The execution time of a regular expression if a string does not match the (x+)+y pattern (the y character is missing at the end).

As you can see, the strings from the first set are processed instantly. However, the processing of the second set increases exponentially! Why so?

The thing is, in the first case a regular expression finds a match on the first try. When processing strings in the second case, everything becomes very complicated. The x+ template can match any number of x characters. The (x+)+ template can fit a string that consists of one or more substrings corresponding with x+. Because of this, there are many options for matching a string with a regular expression. Their number depends on the length of a substring consisting of x characters. Every time the regular expression does not find the y character, it starts checking the next option. Only after checking all of them, the regular expression gives the answer – no matches were found.

The table below shows several possible matches of the xxxx string with the (x+)+y regex:

Fortunately, not all regular expressions are vulnerable to catastrophic backtracking. A regex is vulnerable if it meets the following conditions:

There are two subexpressions, and one of them includes another. Besides, one of the following quantifiers is applied to each of them: '*', '+', '*?', '+?', '{...}'. In the previous example, the (x+)+ subexpression includes x+.
There is a string that can be matched with both subexpressions. For example, the xxxx string may fit both the x+ and (x+)+ templates.

Expressions of the (\d?|....|[1-9])+ type are a small exception. Here the (\d?|....|[1-9])+ expression includes subexpressions \d? and [1-9]. They are enumerated via the '|' operator. These subexpressions can also fit the same string, for example, 111. In this case, applying the '?' quantifier to one of the subexpressions also results in a vulnerability.

Example 2

We found out that the (x+)+y expression is vulnerable. Now let's change it a bit by adding a check for the presence of another character:

(x+z)+y

Now we have the (x+z)+ subexpression, and the xz and xxxxz strings can be matched to this expression. This subexpression includes the x+ subexpression, which can correspond to strings of x, xxxx, etc. As you can see, these subexpressions cannot be matched with the same values. Thus, the second condition is not met and there's no catastrophic backtracking.

Figure 3. Unsuccessful attempt to "break" a regex with a set of strings, each of them corresponds with either the x+ subexpression or the (x+z)+ subexpression.

Example 3

Now let's look at the next regex:

newDate\((-?\d+)*\)

This regex has a task — search for a substring of the newDate(12-09-2022) type. Can we call this regex safe? No. Besides correct strings, the regex will consider correct the newDate(8-911-111-11-11) and even newDate(1111111111111) strings. However, to understand the essence of the issue, such an expression will be enough for us.

None of the above options will lead to catastrophic backtracking. However, it will happen if we process strings of the 'newDate(1111111111111' type.

Figure 4. The execution time of the regex checking strings that do not match the pattern (there is no closing parenthesis at the end of the string)

We see catastrophic backtracking again. It happens because of the (-?\d+)* subexpression, which includes the \d+ subexpression. The '*' or '+' quantifiers are applied to both subexpressions and the same string can be matched with each of them, for example, 111.

Let's compare these observations with the previously examined conditions of the regular expression with a vulnerability:

There are two subexpressions, and one of them includes another. One of the following quantifiers is applied to each of them: '*', '+', '*?', '+?', '{...}'. The (-?\d+)*) subexpression includes \d+;
There is a string that can be matched with both subexpressions. For example, the 1111 string may fit both the \d+ template and (-?\d+)*).

By the way, the newDate$(-?\d+)*$ regex caused a vulnerability (CVE-2021-27293) in a real project – the RestSharp library.

Example 4

As a final example, let's look for vulnerability in a more complex regular expression:

^(([A-Z]:|\\main)(\\[^\\]+)*(,\s)?)+$

The task of this expression is to find strings that represent a list of paths to files or directories. Each element of this list is separated from each other with a comma and a space character. A list item can be represented by a path corresponding to one of two types:

full path, for example:* D:\catalog\subcatalog\file.txt*,
relative path from the main folder, for example: \main\catalog\file.exe.

Thus, the string corresponding to the pattern may look like this:

D:\catalog, C:\catalog\file.cs, \main\file.txt, \main\, project\main.csproj

A regular expression will evaluate such strings without any problems.

The same goes for processing almost any incorrect string processing, for example:

D:*\catalog\file.cs\catalog\file.cs\catalog\file.cs\catalog\file.cs\catalog\file.cs\catalog\file.cs\\\*

However, the situation changes if we pass a string of the following type to a regex:

*D:\main\main\main\main\main\main\main\main\main\main\main\main\main\main\main\\\*

[Figure 5. The running time of the regular expression when processing strings of the *D:\main ...\main\\\* format.

Let's inspect the original regular expression (^(([A-Z]:|\\main)(\\[^\\]+)*(,\s)?)+$) in more details. Note that subexpressions ([A-Z]:|\\main) and (\\[^\\]+)* that follow each other can be matched with the same \main string. Moreover, the following subexpression ((,\s)?) can be ignored, because the '?' quantifier allows the absence of a match with this template.

Thus, it is possible to simplify the original regular expression to check only one special case – strings of the D:\main ...\main format:

^(([A-Z]:|\\main)(\\main)*)+$

The catastrophic backtracking vulnerability becomes clear when we look at the simplified version of this string.

There's a subexpression (([A-Z]:|\\main)(\\main)*)+ with the '+' quantifier. This subexpression includes (\\main)* with the '*' quantifier.
Both subexpressions: (([A-Z]:|\\main)(\\main)*)+ and (\\main)* may fit the same string, for example, \main\main\main.

Therefore, both conditions for a vulnerable expression are met.

Let's highlight the main factors that cause catastrophic backtracking in the ^(([A-Z]:|\\main)(\\[^\\]+)*(,\s)?)+$ regular expression:

The '+' quantifier is applied to the (([A-Z]:|\\main)(\\[^\\]+)*(,\s)?)+ subexpression;
The '*' quantifier is applied to the (\\[^\\]+)* subexpression;
Subexpressions ([A-Z]:|\\main) and (\\[^\\]+)* may fit the same \main string;
The (,\s)? subexpression can be omitted because of the '?' quantifier.

The absence of at least one of them would make the regular expression absolutely safe.

How to protect from catastrophic backtracking?

Let's look at the main ways to protect a regex from catastrophic backtracking. We'll use the newDate$(-?\d+)*$ as an example. The code below is written in C#. However, the similar functionality probably exists in other programming languages that support regular expressions.

Option 1. Add a limit on execution time for processing a string by a regular expression. In .NET, we can do it by setting the matchTimeout parameter when calling a static method or initializing the new Regex object.

RegexOptions options = RegexOptions.None;
TimeSpan timeout = TimeSpan.FromSeconds(1);
Regex pattern = new Regex(@"newDate\((-?\d+)*\)", options, timeout);
Regex.Match(str, @"newDate\((-?\d+)*\)", options, timeout);

Figure 6. The execution time of a regular expression is limited to one second

Option 2. Use atomic groups (?>...):

Regex pattern = new Regex(@"newDate\((-?\d+)*\)", options, timeout);

For expressions marked as atomic groups, the backtracking function is disabled. Thus, of all possible matching options, an atomic group will always be matched with only one substring containing the maximum number of characters.

Although atomic groups are a reliable way of protection against catastrophic backtracking, we recommend to use them carefully. In some cases, using atomic groups can decrease the accuracy of the regular expression's calculation.

Figure 7. Subexpressions marked as atomic groups are no longer vulnerable to catastrophic backtracking.

Option 3. Rewrite a regex by replacing an unsafe subexpression with a safe equivalent. For example, to find a string of the newDate(13-09-2022) type you can use newDate$(\d{2}-\d{2}-\d{4})$ instead of newDate$(-?\d+)*$.

The latter has two subexpressions: (-?\d+)* and \d+. The \d+ subexpression is included in (-?\d+)*. The same substring can match both these subexpressions. The safe equivalent allows matching any substring with only one template due to the mandatory check of the '-' character between templates \d{...}.

Conclusion

Let's sum up:

Regular expressions may be vulnerable to a ReDoS attack, the aim of which is to halt or slow down an application;
The application slows down because of catastrophic backtracking. It occurs if there is a huge number of options for matching an input string with a regular expression and there are no correct options among them;
A regex is vulnerable to catastrophic backtracking if it contains at least one vulnerable subexpression that may cause a large number of matching options.
You can identify a vulnerability in a regular expression by inspecting it for the following conditions:
- There are two subexpressions, and one of them includes another. One of the following quantifiers is applied to each of them: '*', '+', '*?', '+?', '{...}';
- There's a string that may fit both these subexpressions.

And that's all for today :). I hope this article was interesting.

Clean code and successful projects to you! See you in next articles!

What 's new in .NET 7?

Unicorn Developer — Fri, 18 Nov 2022 11:17:33 +0000

.NET 7 is now available, which means that we can enjoy all sorts of new features and enhancements. Let's talk about the most interesting improvements: C# 11, containers, performance, GC, and so on.

C# 11

We have already posted an article and described the novelties in C# 11. In that article, we covered new features: generic math support, raw string literals, the required modifier, the type parameters in attributes, and so on.

By the way, we are already working on .NET 7 and C# 11 support — we plan to add it to PVS-Studio 7.22. The release is scheduled for early December, and here is the link to download the latest version of the analyzer. If you can't wait to try the beta now, feel free to contact us :)

Native AOT

AOT (ahead-of-time) – compilation of the application not into intermediate, but immediately into machine code. Native AOT uses the ahead-of-time compiler to compile IL into machine code while publishing a self-contained application. The dev team of .NET 7 moved native AOT from experimental status to mainstream development. The biggest advantages of native AOT apps are:

startup time;
memory usage;
access to restricted platforms (where JIT is not allowed).

Native AOT applications come with a few limitations:

no dynamic loading (for example, Assembly.LoadFile);
no runtime code generation (for example, System.Reflection.Emit);
no C++/CLI;
etc.

On Stack Replacement (OSR)

In .NET, there is such a thing as tiered compilation. In simple terms, tiered compilation improves the startup time of an application. How? Initially, JIT generates poorly optimized machine code (tier-0) for methods because it simply takes less time. If the number of method calls exceeds a certain threshold, JIT generates more optimized code for this method (tier-1). This approach does not work, for example, with loops, because this can degrade the performance. By the way, at the moment there are only two levels.

OSR allows you to replace the machine code that is currently running with a new, more optimized one. Previously, this feature was only available between method calls. This approach allows you to apply tiered compilation to all methods. This results in quicker compilation with stable performance at the same time. According to Microsoft's tests, the technology helped to speed up the startup of high-load applications by 25%. OSR in .NET 7 is enabled by default for x64 and Arm64.

Central package management (CPM)

Dependency management for multi-project solutions can be a challenge. In situations where you manage common dependencies for many different projects, you can use NuGet's central package management. To get started with central package management, create the Directory.Packages.props file at the root of your solution. Because of CPM, the package version is only specified in Directory.Packages.props, and projects only need to reference the package.

GC Regions

GC Regions is a feature that has been under development for several years. If earlier it was necessary to have several large memory segments (for example, 1 GB), now GC supports many small segments (for example, 4 MB). GC regions provide more flexibility like repurposing regions of memory from one generation to another.

In .NET 7, regions are used by default for 64-bit applications. You can find more details in the .NET GC architect 's article.

Rate Limiting

Rate limiting is the mechanism that limits the access to a resource. You can set a certain access limit, for example, to a database.

To write a limiter in .NET 7, Microsoft added the NuGet System.Threading.RateLimiting package. Most of the work will be done with the abstract RateLimiter class. Here's one of the examples from Microsoft:

RateLimiter limiter = GetLimiter();
using RateLimitLease lease = limiter.Acquire(permitCount: 1);
if (lease.IsAcquired)
{
  // Do action that is protected by limiter
}
else
{
  // Error handling or add retry logic
}

In this case, we are trying to get 1 permit using the Acquire method. Next comes the check – whether permit has been acquired:

if the permit has been acquired, we can use the resource;
if the permit was not acquired, this can be logged or handled as an error.

Built-in container support

Now you can quickly and easily create containerized versions of your applications using the dotnet publish command.

Example:

dotnet add package Microsoft.NET.Build.Containers
dotnet publish --os linux --arch x64 -c Release 
-p:PublishProfile=DefaultContainer

In this case, we add a temporary link to the package to create a container and publish the project for linux x64. The result of executing the commands is an image that will be added to Docker. After that, you can launch the application using the container:

docker run -it --rm -p 5010:80 my-awesome-container-app:1.0.0

You can read more about it here.

Performance enhancements

Performance in .NET is growing year by year. This release was no exception. It would take a separate article just to list all the improvements, so I will only tell you about the most interesting ones.

If you want to know more about the performance improvements in .NET 7, you can read about them here.

Reflection

The reflection overhead is significantly reduced when a call is made more than once for the same item (whether it is a method, constructor or property). Performance has been increased by 3-4 times.

You can read more about improvements of reflection here.

LINQ

In .NET 7, LINQ performance has been improved. For example, the efficiency of the Min and Max methods has been significantly improved when it comes to int and long arrays. This is achieved by vectorizing the processing, namely, using Vector. The results are as follows:

You can read more about LINQ performance improvements in the article.

By the way, it's also very cool that System.Linq was expended with new Order and OrderDescending methods. Previously, when using OrderBy/OrderByDescending, it was necessary to refer to its own value:

var data = new[] { 2, 1, 3 };
var sorted = data.OrderBy(x => x);
var sortedDesc = data.OrderByDescending(x => x);

Now it is not necessary:

var data = new[] { 2, 1, 3 };
var sorted = data.Order();
var sortedDesc = data.OrderDescending();

Regular expressions

At first, I wanted to briefly talk about improvements to regular expressions, but in the course of writing this article I realized that there were too many improvements. Therefore, let me just leave a link to a larger article dedicated to this topic :). It fully describes all the novelties and enhancements of regular expressions. I will also note here that Microsoft has not only improved performance, but also added various functional improvements.

Conclusion

As you can see, there are many improvements in .NET 7. Not all of them are equally useful to all developers, but many technologies will continue to evolve in future .NET releases.

I can single out the following features as the most interesting and useful ones for me:

of course, C#11;
on stack replacement (OSR);
Central package management (CPM);
GC Regions.

The article, of course, did not list all the new features, but only the most interesting ones (in our opinion). You can find all the improvements here.

Have you already tried any of the new .NET features? Feel free to share them in the comments.