Proper separation in layer architecture seems more of a theoretical goal than reality. Any strategies around that?

Question

I am currently refactoring an application of mine I wrote back in 2015. Things were done I am not proud of, constant learing process and all that.

Previously everything was mashed together in one executable (database calls in forms code shudder), this should now of course be different. The app in question is a windows forms application talking to a MySQL database which is also used for some php pages. Think course organisation with giving members the ability to sign up and put their name forward for attending courses.

Now, by summarising what I have read so far, having different layers in your application helps with readability, testability and maintainability. I get that and generally like the approach of separating different concerns. The problem™ I am currently facing (and this might be just me overreacting and procrastinating) is, that having layers with clearly defined boundaries does not seem to fully disentangle needed references and requirements from one particular layer from other layers.

The tiers of my application are as thus:

1. DataModel: This contains my domain model / all relevant (rich) data classes. Business logic lives here (e.g. what happens with an application when the course is already full, validation, ...)
2. DAL: This defines interfaces to the persistence layer. Most notably one interface that includes the interfaces for all externally relevant repositories
3. Concrete persistence implementation: All repositories are only exposed through their respective interfaces
4. BusinessLayer (Application- or ServiceLayer would probably be more appropriate): Interaction between the outside world and my domain model lives here. I define "Handlers" to manipulate my data and trigger the persitstence mechanisms of it. I can throw exceptions if a validation fails before going on, so that a "consuming" application can interact with the user to get the data right.
5. UI: Here everything sort of comes together. Different forms are presented to the user depending on what they want to do.

This seems (maybe only at a first glance) to be rather excessive, but I feel rather comfortable with the separation. BUT. Looking at which tier needs to include references to what, things start to seem a bit more tangled than I would have expected from "separating everything into different layers":

• DAL: depends on DataModel
• Concrete persistence: depends on DAL and DataModel
• BusinessLayer: depends on DataModel and Concrete persistence (and thus on DAL) and DAL
• UI: depends on BusinessLayer, DataModel, DAL (because interfaces and dependency injection) but also on a comncrete persistence implementation.

What bugs me with this is n-fold:

1. DAL and concrete persistence would not usually use any of the "rich" functionality from my domain model classes (validation, special handling of added elements), they get the correct state of the data from the database. Also the UI should never be able (thing rogue programmer personality) to directly manipulate my data objects, changes should always run through my Business Layer. Would it make sense to define the properties of the classes (basically database fields now) only as interfaces in the "DataModel" and work with the interfaces throughout? The rich functionality can then be added in concrete implementations inside the Business Layer, which I can construct by passing the interface to a constructor. The downside would be that I would need to implement the interface in every part that actually uses it, so Business Layer, concrete persistence and UI.
2. Speaking of "the UI should never be able to": Because of dependency injection (and other circumstances like reverting to a local cache-repository in case of a network outage) the UI as entrypoint to the application needs to define which concrete repository it wants/needs to use. So far so good. But now I have direct access to the repositories exposed in my UI Layer although I only need it there to pass it on to the Business Layer which is supposed to do all of the persistence triggering. I suppose there is no short and elegant way around it?
3. I have no specific View Model and am not sure I need one. Short of the actual database IDs (and even they are needed to identify the objects being manipulated) almost all properties of my data classes are going to be displayed and potentially edited on the UI anyway, so it rarely makes sense to construct specific classes intended for a specific view. If I were to do so I would have no idea where to define them. If I define them in the UI, then my business logic needs to rely on the UI (which would constitute a circular reference which at least Visual Studio tends to dislike). If I define them externally (as interfaces even) I would start to swamp my Business Layer with presentation specific requirements. How is this normally done?
4. Speaking of swamping the Business Layer with presentation specific requirements: How can one prevent that requirements for one layer spill over to other layers? I guess sometimes it is inevitable, but: On my UI I want to display comboBoxes to quickly select the person to be edited (the ones holding a course as well as the ones attending one). Because the comboBox does not need to show things like addresses but only title, first and last name (and provide the database id of the selected entity) I do not need to do a full SELECT * FROM and return very rich objects (yes yes small data compared to video streaming yadda yadda, but I think it pays off to be somewhat frugal with resources). Thus a requirement from the UI side was born "we need a dummy person entry". This of course trickled down to the business layer where my handler for each category of persons to be edited now has a function to retrieve such a list of dummy entries. Naturally this function needed to go one step deeper into my DAL and concrete implementation. And because these functions are supposed to return something, I now have a class for a dummy entry in my data model. This class has no business logic attached and its data is not persisted, it just sits there. Have I missed an obvious solution aside from "just read the whole list of full datasets and pass that to the UI and be done with it"? On a sidenote: Who is responsible for defining validation constraints like "This field should be 255 long at most"? I can't do it in the interfaces, because interfaces. If I do it in the concrete implementation in the Business Layer, how do I get the information across to the person writing a new persistence repository (except telling them and documenting it outside of the code)?

Argh... This got way longer that I thought, but at least it had the cathartic effect of allowing me to (hopefully) precisely describe my problems to the world an me and get my thoughts in order. I can try to fill this with actual code if necessary, but I think the underlying concepts do not really depend on code. Thank you for bearing with me and answers to my n buggings or general recommendations and thoughts are highly appreciated.

Answer 1

2.Speaking of "the UI should never be able to":
I suppose there is no short and elegant way around it?

I would say that there is one. If you want to go purist (which is not actually always needed, quite often the limitation by convention is just fine) - you can extract your presentation level code in appropriately typed library (for example Win Forms one) which will depend only on contracts of other layers and then have a separate composition root project which will reference all the needed parts and start the actual app. This can involve a little bit more ceremony but in my experience usually not that much.

P.S.

BusinessLayer: depends on DataModel and Concrete persistence (and thus on DAL)

This is a curious dependency. From your explanations of levels it seems that BusinessLayer should only depend on DAL (i.e. layer/project containing data access contracts)

Answer 2

That is an interesting list of questions. I can answer all of them, but it'd be a rather long answer. Instead, I'm going to point out what I think are the keys for you to figure it all out.

Most of the issues and questions you have come from your chosen architecture: a layered architecture.

I would propose that you move to the following approach:

1. Use CQRS, meaning, separate the Commands (operations that mutate state), from Queries (operations that just return data, without mutating state)
2. On your Command side, use an "onion" approach instead of layered: Clean Architecture, Hexagonal Architecture, Ports and Adapters.

Here are few benefits of using this approach:

• Separating commands and queries allows you to solve different problems with their own solutions. For example, regarding your Combobox question. You have a requirement which could be to get a list of suggested people when the user starts typing. For this, I'd just define a query SearchText and the result would be a list of objects with Id and Name. The query handler implementation would be a straight query to the DB which would do the right query. As the query defines the inputs and outputs, you are not forced to return full domain objects. You can return exactly what is required for each Query use case.

• The separation then allows for solving the challenges of commands with a completely different approach. My preferred approach is Ports and Adapters. With this approach, the Command is a Driver port. It's defined by the Core to allow an external Actor to trigger an operation in the system. The Command defines the exact inputs that the use case requires, you don't need to pass full domain objects. The core then will implement the business logic and it will define the driven ports that it needs. This is very important. It's not the DAL that defines a bunch of interfaces that the Core might or might not need. It's the core that defines the exact interfaces that it needs and it couldn't care less what implements them. It's then the Adapters that implement all the driven ports that the core defines. The Core does not depend on anything. It's the outside layers that depend on the inside ones.

You should be able to derive the rest of the answers from the pointers above, otherwise, let me know in the comments.

Another point: you find it suspicious that a UI requirement affects the business logic, but it's a matter of perspective. First of all, it's not a UI requirement. It's a business requirement and as with any business requirement, you might have to change multiple parts of the system. On the other hand, there's the misconception that application layers are there to decouple the UI from the business logic and the business logic from the data access, but that is not correct. No matter how many layers and abstractions you add, those three things will always be coupled. If you need to add a new field to a form, you'll have to add it all the way down to the DB, regardless of the layers. The only decoupling that you can do is horizontal (for example, changing Form A shouldn't cause changes on Form B).

Another point: you say that the UI is the entry point of the application and this is also a misconception. You might have your forms on the same project where your entry point is, but that's just a choice. Your entry point sets up the DI composition root. The composition root is what references all interfaces and all implementations. The rest of the pieces of the puzzle can get away by only referencing interfaces. For example, your UI only needs a dependency on the Commands and Queries that need to dispatch.

Answer 3

Although it can be sympathized to your tiers of your application, I fundamentally believe that Onion architecture(Clean architecture) and traditional "N-Layer" architecture applications should be considered as revised ways to develop application.

Any UI framework can be used at UI layer.

Traditional "N-Layer" architecture applications

These layers are frequently abbreviated as UI, BLL (Business Logic Layer), and DAL (Data Access Layer). Using this architecture, users make requests through the UI layer, which interacts only with the BLL. The BLL, in turn, can call the DAL for data access requests. The UI layer shouldn't make any requests to the DAL directly, nor should it interact with persistence directly through other means. Likewise, the BLL should only interact with persistence by going through the DAL. In this way, each layer has its own well-known responsibility. Read more here

Onion architecture(Clean architecture)

Clean architecture puts the business logic and application model at the center of the application. Instead of having business logic depend on data access or other infrastructure concerns, this dependency is inverted: infrastructure and implementation details depend on the Application Core. This functionality is achieved by defining abstractions, or interfaces, in the Application Core, which are then implemented by types defined in the Infrastructure layer. A common way of visualizing this architecture is to use a series of concentric circles, similar to an onion. The next figure shows an example of this style of architectural representation. Read more here